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SUMMARY 


PROBLEM 

A  research  objective  of  the  Air  Force  Human  Resources  Laboratory  is  the 
development  of  methods  for  defining  the  components  of  innovative  technology 
and  for  measuring  the  effects  cf  the  incoming  technology  on  Air  Force  human 
resources.  The  human  resource  parameters  of  concern  include  manpower  (e.g., 
numbers,  job  types,  skill  levels),  training  considerations,  and  cost  data. 
Earlier  Investigations  established  the  feasibility  of  using  human  resources 
data  in  design  trade  studies  and  of  determining  and  graphically  depicting  the 
array  of  trade-off  options  available  before  Inception  of  hardware  design.  The 
purpose  of  the  review  reported  here  was  to  establish  the  current  status  of 
the  methodology  for  forecasting  and  assessing  technology  and  for  quantizing 
human  resource  parameters  with  respect  to  the  impact  of  Incoming  technologies. 

APPROACH 

A  search  and  critical  analysis  of  the  literature  was  undertaken  to  review 
the  status  of  forecasting  and  assessing  technology  and  of  techniques  for  pre¬ 
dicting  the  Impact  of  technology  on  human  resource  parameters.  The  size  of 
the  literature  base  in  the  area  of  technological  forecasting  and  technology 
assessment  precluded  the  conduct  of  an  exhaustive  review.  Rather,  the  deter¬ 
mination  was  made  that  inclusion  of  selected  references  which  described 
current  methodologies  in  detail  and  which  were  judged  as  representative  would 
better  satisfy  the  objectives  of  the  research  effort.  Emphasis  was  directed 
toward  location  of  quantification  methods  for  human  resources  impact  of 
technological  innovations. 

Computer-based  reference  systems  of  the  Defense  Documentation  Center, 
National  Technical  Information  Service,  Systems  Development  Corporation  and 
The  Ohio  State  University  were  employed  as  integrants  to  a  conventional 
search  of  the  professional  literature. 

RESULTS  AND  CONCLUSIONS 

A  total  of  140  documents  applicable  to  this  effort  were  reviewed. 

Seventeen  of  these  were  identified  as  having  major  significance  to  the  present 
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effort.  For  these  17,  3-page  expanded  summaries  were  generated  and  appended 
to  this  report  of  the  results  of  the  literature  survey. 

Review  of  the  literature  failed  to  provide  solution  to  the  problem  of 
quantizing  human  resource  parameters  with  respect  to  Impact  of  incoming  tech¬ 
nologies.  However,  the  use  of  a  normaclve  forecasting  technique  was  strongly 
supported  by  the  literature.  In  particular,  a  relevance  tree  approach  was 
the  technique  viewed  as  amenable  to  the  problem  of  successive  identification 
of  increasingly  finer  components  In  an  organized,  structured  manner.  The 
relevance  tree  procedure  known  as  a  Design  Option  Decision  Tree  was  Identified 
as  practical  for  detailing  a  system  to  a  level  permitting  the  Identification 
and  assessment  of  human  resource  components  for  Impact  quantification. 

Development  of  quantizing  methods  and  procedures  for  evaluating  the 
adequacy  of  the  evolved  techniques  Ir.  generating  human  resources  data  for 
system  designers  and  personnel  planners  are  highlighted  as  goal  directions 
for  the  remaining  study  effort. 
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Section  I 
INTRODUCTION 

The  Increasing  demand  to  Improve  the  capabilities  of  new  weapon  systems 
entering  the  inventory  of  the  military  is  forcing  a  change  In  the  traditional 
conceptualization  of  the  man-machine  Interface  and  the  support  requirements 
of  a  system.  Advanced  manpower  planning  and  restrictive  budget  constraints 
»  require  the  systems  planner  to  refine  the  projected  human  resource  estimates 

to  result  in  more  optimal  employment  of  the  human  resource  component.  It  is 
no  longer  sufficient  to  roughly  approximate  the  human  resource  requirements 
necessary  for  operating  and  maintaining  a  given  weapon  system.  Since  a  Riven 
weapon  system  usually  integrates  technological  innovations,  It  becomes  essential 
to  be  able  to  predict  the  Impact  of  these  innovations  on  human  resource  param¬ 
eters.  Military  management  must  therefore  develop  a  feasible  way  to  locate 
and  anticipate  the  advent  of  technological  innovations  and  predict,  in  advance, 
the  human  resource  consequences. 

Historical  evidence  that  such  an  action  is  necessary  when  dealing  with 
complex  weapon  systems  can  be  found  readily  in  prior  Air  Force  procurements. 

A  single  anecdotal  reference  la  used  for  Illustrative  purposes.  In  the  intro¬ 
duction  of  the  radar  gunslght  on  the  F-86  fighter  in  Korea,  maintenance  aspects 
of  the  subsystem  were  not  given  sufficient  advance  consideration.  Consequently, 
the  sight  became  operational  in  the  field  but  could  not  be  adequately  main¬ 
tained.  A  priority  effort  was  undertaken  to  train  existing  gnnsight  repairmen 
on  the  new  system,  a  logical  decision,  since  the  additional  training  would  be 
minimal  for  this  group  of  personnel.  Unfortunately,  the  training  was  conducted 
•  at  the  manufacturer's  plant,  an  action  which  effectively  removed  conventional 

gunslght  repairmen  from  the  field,  further  compounding  an  already  serious 
maintenance  problem.* 

In  the  present  environment,  a  persuasively  compelling  argument  for  re¬ 
flecting  the  impact  of  technological  innovation  on  human  resources  in  the 
design  phase  is  found  in  cost  considerations.  The  largest  single  life-cycle 
cost  in  a  weapon  system  is  the  human  resource  cost  over  the  period  of  time 

*  Personal  observation  of  the  senior  author. 
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Lho  system  is  In  the  operational  Inventory.  A  system  designer  must  have 
access  to  data  concerning  the  human  resource  parameters  relative  to  various 
design  options  in  order  to  select  the  most  effective  design  based  on  life- 
cycle  cost  considerations. 

Eckstrand,  Askren,  and  Snyder  (1967)  emphasized  this  need  for  a  detailed 
consideration  of  human  resources  by  classifying  these  resources  as  an  "impor¬ 
tant  design  parameter"  but  cautioned,  "unfortunately,  the  data  and  technology 
for  (this)  involvement  .  .  .  have  not  been  developed  to  the  required  degree." 

In  this  paper  the  term  technology  will  be  used  to  describe  the  area  of 
knowledge  that  composes  the  applied  techniques  used  in  the  production  of  a 
weapon  system.  Therefore, technology  would  encompass  all  disciplines  involved, 
e.g.,  mechanics,  electronics,  and  metallurgy.  An  innovation  will  be  considered 
as  any  new  aspect  of  a  given  discipline  which  has  potential  practical  applica¬ 
tion.  What  we  are  therefore  seeking  is  a  way  to  predict  the  impact  of  a  tech¬ 
nological  innovation  upon  the  human  resource  requirements  of  a  specified  weapon 
system.  If  one  conceptualizes  the  process  generally  as  was  done  in  Figure  1, 
it  can  be  seen  that  the  design  process  is  viewed  as  making  decisions  based  on 
the  available  technology  to  meet  the  objectives  established.  The  objective 
may  be  singular  or  multiple  so  that  the  design  process  is  restrained  by  exist¬ 
ing  or  emerging  technology  and  the  pre-established  design  objectives.  We 
are  hypothesizing  that  the  availability  of  a  technological  Innovation  would 
affect  the  design  process.  If  the  innovation  is  implemented  in  the  design, 
the  result  will  be  a  different  weapon  system  design.  Each  weapon  system  de¬ 
sign  may  have  different  human  resource  requirements. 

The  simplified  diagram  presented  in  Figure  1  illustrates  the  basic 
considerations  being  addressed.  Naturally,  an  Innovation  may  be  rejected  as 
having  no  impact  on  design.  Or  the  selection  of  the  innovation  may  have  such 
a  small  Impact  on  the  design  process  that  no  significant  difference  would  be 
reflected  in  the  weapon  system  produced.  It  would  also  be  possible  to  have 
two  weapon  system  designs  which  would  have  the  same  human  resource  require¬ 
ments.  (Note:  in  this  paper,  the  human  resource  includes  not  only  the 
operator  of  the  weapon  system  but  also  the  support  staff  which  maintains  the 
total  man-machine  system.) 


f. 
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Figure  1.  Technological  Innovation  Impact  on  Human  Resource  Requirements 


The  final  goal  would  be  the  establishment  of  a  technique  which  would 
allow  a  quantifiable  description  of  an  Innovation  which  provides  the  capa¬ 
bility  of  predicting  human  resource  requirements.  This  action  would  effectively 
prelude  the  detailed  design  process  and  weapon  system  outcome  stages  and 
establish  the  link  between  technological  Innovation  and  human  resource  parametei's. 

The  present  research  effort  is  a  multiphased  program  directed  toward 
the  generation  of  methods  for  identifying  and  measuring,  prior  to  system 
development,  the  effects  of  technological  innovation  on  the  human  resource 
parameters  required  to  maintain  and  operate  a  weapon  system  which  will  in¬ 
corporate  the  technological  innovations  into  its  design.  The  ideal  objective 
of  this  effort  is  to  generate  this  data  in  forms  useful  to  system  designers  and 

personnel  planners  to  be  considered  in  design  decisions  and  manpower  require¬ 
ment  projections  (see  Appendix  A-1) . 

Presented  in  the  following  pages  are  the  results  of  the  first  phase  of 
the  effort,  a  search  and  critical  analysis  of  the  literature  reviewing  the 
current  status  of  forecasting  and  assessing  technolcgy  and  in  the  techniques 
for  predicting  the  impact  of  technology  on  human  resource  parameters.  This 
survey  is  not  an  exhaustive  review  of  the  literature  in  the  area  of  techno¬ 
logical  forecasting  and  technology  assessment.  It  was  felt  that  the  objectives 
of  the  study  would  be  better  served  by  the  Inclusion  of  selected  references 
which  discussed  the  current  methodologies  in  detail  and  which  were  judged  to 
be  representative  of  the  field.  For  a  more  comprehensive  grasp  of  this  field, 
the  reader  is  referred  to  Martino  (1972),  Ayres  (1969),  or  Cetron  (1969).  In 
addition  to  the  material  contained  in  these  references,  a  reader  Interested 
in  a  more  extensive  consideration  of  technological  forecasting  is  referred  to 
the  bibliographies  provided  by  Harrison  (1971),  800  references,  and  to 
Chamberlain  (1971),  108  references. 

A  total  of  140  documents  applicable  to  this  effort  were  reviewed.  Of 
these  documents,  17  were  identified  as  being  of  major  importance  to  the  present 
effort.  F.xpanded  summaries  were  made  of  these  key  documents.  These  summaries 
an*  included  as  an  appendix. 


Fmpliasls  oi  this  survey  was  directed  toward  identification  of  melliods 
for  assessment  of  the  impact  on  human  resource  parameters  ol  advances  in 
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technology.  Hany  of  the  usable  and  relevant  documents  described  efforts  per¬ 
formed  by  or  for  the  various  branches  of  the  military.  Air  Force  sponsored 
studies  predominated,  a  finding  comparable  to  an  earlier  review  of  the  litera¬ 
ture  on  prediction  of  personnel  requirements  (Folley,  Falrman  and  Jones,  1960). 

In  addition  to  a  conventional  search  of  the  professional  literature,  the 
principal  sources  for  document  Identification  were  the  computer-based  reference 
systems  of  the  Defense  Documentation  Center  (DDC) ,  National  Technical  Informa¬ 
tion  Service  (NTIS),  the  System  Development  Corporation  INFORM  and  ERIC/ORBIT, 
and  ihe  Ohio  State  University  computer -search  systems. 
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Section  II 
FINDINGS 


methodologies  for  predicting  ato  assessing  tecnnology 


ITT  r'""  «-PHsHaE  E,  cUs,IF,i„*  Lhe  Lechnl,o.A 

:::“iZur  rrr^ ---  -  -- 

^He  oEJecLIvA  „F  .„  F UTZcVL^r"*"  ““ 

f  ,  Eorecasc,  which  Is  always  to  predict  the 

i“"’Tr;”'  r *" 

.  ThA  RAAAral  concApt  OF  what  Ia  tAgardAd  aa  tAchnology  mast  also  ba 

AdMtl'"  f  «=lRntlFlo  knowlAdgA  Into  a  tAchnoIogy  or  thA 

on  o  an  Axlatlng  tachnology  to  a  now  usa  ara  two  distinct  procassAs 
which  should  be  addressed  Tn  »-ka»  e^ 

transFcf  In  th  «  ‘he  concarn  Is  with  vattlcal 

•nsFar.  In  tha  sacond.  horltontal  transFar  (Roonay,  1,71).  Whatavar  tha 

o  ncarn.  tha  Impact  oF  tachnologlcal  changa  .oat  ba  avalnatad  In  tha  co  tart 

O  tha  ontcona  objactlva.  Tha  oblactlva  .ay  ba  to  astabllah  tha  Faaslblll 
Of  some  Innovation  or  to  resolve  a  u,  sioiiity 

«  decision  problem.  Due  to  the  multi- 

dlsclpllnad  davalop„ant  oF  this  araa  an  unwlaldy  conglonarata  oF  .athodologlas 
anargad  which  ara  nsad  to  projact  tha  Fntnra.  As  UnFord  (1,77,  pnilT" 


The  consideration  of  the  subject  of  technoloelcal 
recasting  has  been  made  more  difficult  by  the 
to^siL^?*  terminologies  that  applied  to  the  same  or 
i  r  methodologies,  by  the  proliferation  of 
methodologies,  and  by  the  discussions  of  'Intuitive  ' 
ornM  'normative'  techniques.  Further 

retiree  aU®  considering  techniques  of 

resource  allocation  as  forecasting  methods. 

In  attamptlng  to  addrass  this  probla.,  so.a  taxonowlc  structura  Is  ragulrad 

arhaps  tha  .ost  parsl.„nlo„s  attack  would  ha  to  build  upon  tha  assumptions' 
Of  a  technological  forecast.  umpcions 


Sln,.a  dlract  knowladga  oF  Future  avants  Is  unknown,  thara  would  appear  to 
■■  primary  asaump,  l„„s  undarlylng  a  Foracast  h,R  mathodolugy .  Kic,,  j,, ,, 

. . .  —ran  ba  ohtalaad  .and  . .  si!:. 
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there  Is  some  relationship  between  past  and  future  events.  The  difference 
in  methodological  approaches  therefore,  largely  hinges  upon  th’  type  of  re¬ 
lationship  assumed  between  past  and  future  events. 

A  popular  initial  taxonomy  includes  three  major  methodological  approaches: 
Intuitive,  exploratory,  and  normative.  The  current  statu  of  the  discipline  in¬ 
dicates  that  these  categories  are  not  satisfactory.  Several  authors  have 
pointed  out  that  all  approaches  can  be  considered  intuitive  (Martino,  1972, 
Lanford,  1972a).  The  term  intuitive  therefore  loses  its  function  when  con¬ 
sidered  Ir  the  broader  philosophical  sense.  The  approach  normally  labeled  ex¬ 
ploratory  also  has  lost  much  of  its  value  since  so  many  approaches  fall  under 
this  ca’Tgoiv.  iherefore,  a  new  taxonomy  seems  to  be  appropriate.  It  is  sug¬ 
gested  that  there  are  basically  two  major  approaches,  that  of  projecting  the 
future  from  available  data  (extrapolation),  and  that  of  establishing  a  set  of 
desired  goals  in  the  future  and  working  back  to  the  present  situation  to 
determine  the  changes  necessary  to  obtain  the  goals  (normative) . 

In  addition  to  these  major  approaches,  it  is  beneficial  to  consider  the 
technique  for  gaining  the  basic  data  used  in  the  projections.  There  appears  to 
be  three  basic  techniques  employed  in  the  extrapolation  approach;  judgmental, 
projection  and  analogy.  In  the  judgmental  technique,  the  expert  opinion  of 
individuals  is  collected  in  some  fashion  to  establish  a  quantifiable  value. 

In  the  projection  technique,  some  relatively  empirical  measure  is  collected, 
and  then  a  curve  is  selected  to  project  future  expected  data  points.  In  the 
analogy  technique,  some  associated  variable  is  linked  to  the  variable  of 
interest  and  its  patterned  relationship  is  used  to  determine  the  outcome. 

The  normative  approach  appears  to  consist  of  two  basic  methodologies; 
network  construction  and  matrix  consideration.  In  the  network  construction 
approach,  points  of  critical  concern  are  identified  projecting  backwards  from 
desired  goals  to  the  current  situation.  In  matrix  consider  .at  ion,  several 
vectors  of  interest  are  generated  and  a  matrix  of  interactions  is  developed  to 
establish  a  quantifiable  value  of  each  interaction  outcome.  Figure  2  displays 
the  taxonomy  with  some  specific  methodologies  displayed.  [For  a  more  compre¬ 
hensive  presentation  of  methodologies,  see  Lanford  (1972a).  who  has  collected 
all  the  methodologies  together  for  review. ] 


Judgmental  Forecasting 


Although  this  paper  considers  the  judgmental  technique  as  an  extrapolation 
method,  it  is  usually  considered  exploratory.  The  classification  is  made  on 
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the  basis  that  these  procedures  are  concerned  with  exploring  the  possible 
futures  inherent  in  present  capabilities.  Though  all  methods  discussed  up 
to  this  point  are  exploratory,  the  term  "exploratory  forecasting"  in  recent 
popular  usage  appears  to  be  frequently  associated  with  the 'Delphi  procedure. 

Since  its  reported  genesis  in  1963  (Dalkey  and  Helmer) ,  the  Delphi 
procedure  has  been  extensively  used  by  those  engaged  in  exploratory  forecast¬ 
ing  exercises.  This  popularity  may  be  based  on  the  fact  that  the  Delphi  ap¬ 
proach  is  designed  to  arrive  at  a  prediction  of  what  the  future  situation 
will  be,  rather  than  a  consideration  of  the  many  possible  situations  that 
could  exist  in  the  future. 

A  second  basis  for  appeal  of  the  Delohi  procedure  may  lie  in  its 
demonstrated  reliability.  Reliability  heri^  is  used  in  the  experimental  sense 
comparable  results  can  be  produced  by  repetition  of  the  procedure  with  dif¬ 
ferent  panels  (Ayres,  1969). 

The  Delphi  procedure  has  characteristics  which  serve  to  differ¬ 
entiate  it  from  other  tools  of  the  forecaster.  It  maintains  anonymity 
of  the  participant  experts  throughout  the  complete  cycle,  thus  obtaining  the 
advantages  possible  from  group  Interaction  and  at  the  same  time  obviating  the 
disadvantages  Inherent  In  such  an  approach:  the  effects  of  vested  interests, 
common  biases,  disproportionate  Influence  of  a  dominant  individual,  and 
pressure  to  agree  with  the  majority  view  (Gordon  and  Helmer,  1964). 

The  Delphi  procedure,  however,  should  not  be  viewed  as  the  complete 
panacea  to  the  forecasters'  dilemma;  for,  as  cautioned  by  Martino  (1972), 
"Although  the  Delphi  method  ...  is  both  popular  and  practical,  it  suffers 
from  the  shortcoming  that  it  is  almost  completely  subjective  in  nature." 

A  further  criticism  of  the  Delphi  procedure  is  that  it  concentrates  on  one 
event  at  a  time  and  thus  does  not  allow  a  consideration  of  Interactions 
among  the  events  being  forecasted.  Reacting  to  this  criticism,  a  procedure 
was  devised  by  Helmer  and  Gordon,  termed  "Cross  Impact  Analysis,"  which  can 
be  used  to  arrive  at  an  analysis  of  the  interactions  among  events  forecast 
by  a  Delphi  panel  or.  for  that  matter,  from  any  other  source  of  forecast. 
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Lanford  (1972b)  discusses  the  impact  study  approach  and  relates  it  to 
the  system  analysis  approach.  Blackman  (1973)  compares  a  cross  impact  model 
against  a  decision  tree  approach  for  long-range  planning  and  asserts  that 
the  cross  impact  analysis  has  greater  utility  in  that  it  reduces  the  com¬ 
plexity  of  the  forecasting  task. 

Prolection  Forecasts 

In  this  method  of  forecasting,  the  forecaster  systematically  records 
the  past  rate  of  technological  Innovation  and  advance  in  the  area  of  Interest, 
assumes  this  past  rate  will  continue  into  the  future  and  fits  an  appropriate 
trend  to  the  data,  generally  through  the  use  of  a  regression  technique.  An 
example  of  application  of  this  procedure  can  be  found  in  Lamar  (1969).  This 
method  of  forecasting  is  subject  to  the  same  caveats  that  are  expressed  for 
statistical  applications  of  regression  analysis;  for  example  see  Ostle  (1963). 

A  more  complex  approach  is  to  select  a  curve  fitting  technique  employed 
in  other  areas  of  research.  This  approach  to  forecasting  has  made  use  of 
syateraatic  procedures  developed  by  biologists  to  explain  and  predict  the 
process  of  biological  growth.  The  basis  for  this  adoption  lies  in  an  observed 
parallelism  between  biological  growth  and  "growth"  in  performance  o'"  echno- 
loglcal  devices.  The  biological  growth  curves  have  been  used  as  conceptual 
models  for  technological  development  with  the  goal  of  obtaining  greater  ac¬ 
curacy  of  prediction,  i.enz  (1962)  described  this  approach,  but  did  not  lend 
strong  support  to  It.  Biological  growth  curves  based  on  mathematical  functions 
have  also  been  investigated  by  forecasters.  Two  growth  curves  frequently  used 
in  technological  forecasting,  the  Pearl-Reed  curve  and  the  Gompertz  curve,  are 
discussed  by  Martino  (1972).  Martino  contends  that  neither  method  can  be 
viewed  as  being  completely  ilgorous.  However,  he  does  conclude  that  the  use 
of  growth  curves  is  pteferable  to  the  Delphi  method  if  sufficient  historical 

data  are  present. 

Analogy  Forecast 

Analogy  forecasting  is  an  attempt  to  arrive  at  a  forecast  by  comparing 
the  present  situation  with  some  similar  technology  in  the  past.  A  more 


precise  attempt  in  this  area  would  be  the  establishment  of  a  relationship  with 
another  innovation  or  variable  which  can  be  measured  or  quantified  so  that  if 
the  relationship  is  knowrii  a  projection  can  be  made  basi'd  on  the  analop.uus  in¬ 
dicator.  An  example  would  be  the  work  of  Maxlish  (196"))  who  used  the  develop¬ 
ment  pattern  of  railroads  as  an  analogous  indicator  for  the  space  program  In 
exploring  the  process  of  projection  by  historical  analogy.  A  number  of  problems 
are  associated  with  this  technique.  One  of  the  major  problems  is  referred  to 
as  the  error  of  "causal  analogy"  (O'Conner,  1970).  This  error  is  committed 
when  one  observes  a  few  similarities  between  the  past  technology  and  the  situ¬ 
ation  he  is  seeking  to  forecast  and,  without  gathering  more  evidence,  concludes 
that  the  two  situations  are  in  fact  analogous.  More  involved  problems  in  fore¬ 
casting  by  analogy  Include  the  introduction  of  possible  forecaster  bias  in  his 
refusing  to  consider  "implausible"  outcomes.  An  associated  problem  is  that  of 
historical  uniqueness.  In  this  problem  area,  the  forecaster  is  drawn  into  an 
erroneous  conclusion,  since  his  historical  base  is  not  representative  of  the 
past  situation.  These  sources  of  error,  as  well  as  others,  are  discussed  in 
more  detail  in  Mazlish  (1965). 

Network  Construction 

The  techniques  of  forecasting  grouped  under  the  descriptor  "network 
construction"  are,  to  a  large  extent,  products  of  systems  analysis  procedures 
and  emphasize  the  use  of  some  quantitative  approach.  Cetron,  Martino  and 
Roepcke  (1967)  report  on  30  quantitative  methods  applicable  to  research  and 
development  decisions.  Normative  methods  attempt  to  identify  and  to  locate, 
with  reference  to  one  and  another,  the  elements  of  some  system  and  to  investi¬ 
gate  relationships  existing  between  the  elements  within  the  system.  These 
techniques  are  amenable  to,  and  are  frequently  used  for,  examination  of  system 
limitations,  capabilities,  costs,  and  design  optimizations  for  identifying  an 
approach  leading  to  best  attainment  of  a  special  goal.  Three  basic  techniques 
of  network  construction  forecasting  which  are  the  most  typically  used  in  fore¬ 
casting  exercises  are:  relevance  trees,  mission-flow  diagrams,  and  morphological 
models. 

Morphological  models  are  used  in  the  unique  circumstance  where  no 
hierarchical  relationships  exist  between  the  elements  being  considered  and 
where  each  element  can  be  altered  without  ensuing  system  ramifications 
(Zwicky,  1969). 
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I  111-  ii,;.-  ol  mission-!  low  diaKr.tms  Is  most  practical  for  system  descrip¬ 
tion.  The  system  is  described  in  terms  c^f  branching  paths  where  the 
description  can  be  accomplished  by  use  of  one  or  more  series  of  segmented 
steps  or  paths.  This  technique  was  originated  to  ]-ermit  the  analysis  of 
the  elements  of  selected  military  missions.  Probability  of  accomplishment 
or  cost,  as  examples,  can  be  associated  with  each  step  along  the  flow 
diagram,  and  the  diagram  can  then  be  esed  to  generate  a  normative  forecast 
for  another  variable  such  as  performance  requirements  to  satisfy  the  mission. 

The  relevance  tree  approach  is  typically  used  in  a  circumstance  where 
specific  levels  of  abstraction,  or  complexity,  .-an  be  Identified.  Hlstor:*- 
cally,  the  relevance  tr.e  metluxl  w  is  Ci..mnonly  used  for  ori.anized  description 
purposes  and  iiad  wide  biological  an.l  botanical  application.  The  initial 
application  of  a  relevance  tree  approach  to  forecasting  was  apparently  per¬ 
formed  bv  Wells  in  1958  (tletron,  19b9). 

A  full-si'ali>  application  of  the  reli'vance  tree  method  was  performed  by 
bscli  for  the  Honeywi'l  I  Corporation.  Cominuing  development  by  Esch  and 
destice  resulted  in  a  procedure  lalle.i  "I’laiming  Assistance  Through  Technical 
Evaluation  of  Relevance  Numbers  (PATTERN)"  (Esdi,  1965).  In  this  approach 
"relevance"  numbers  are  assigned  to  the  options  at  each  level  of  the  tree. 

A  path  is  selected  through  the  tree  and  the  produi-t  of  the  numbers  on  each 
branch  of  the  path  fi’l  lowed  is  obtained.  This  product  is  used  as  a  figure 
of  merit  for  comparison  ot  tliis  p.ith  against  other  possible  paths  in  the 
tree.  At  each  level  in  the  trei',  I  in*  relevance  numbers  assigned  are  pro¬ 
portions  which  satis! V  a  restriction  that  the  total  for  the  level  must  sum 
to  unitv.  This  process  is  referred  to  as  "normalization"  of  the  relevance 
numbi’rs  and  is  .1  d  ist  ini’u  i  sh  ing  feature  ol  the  I’ATTERN  approach  and.  In 
g.eni-ral,  of  the  ap[)lication  ol  relevance  numbers  1.  reli'vanci'  trees.  A 
danger  in  following  this  tei-hnique  I  ie  ;  in  pli'i,];'  iio  much  confidence  in 
the  significance  ot  differences  between  tie-  numbers  obtained  tbrough  such  a 
procedure  (Martino,  1972)  (see  Appiuidix  A-2).  As  an  example,  it  can  be 
established  quickly  that  the  figure  of  merit  for  any  specific  approach  in  a 
simple  tree  having  only  two  levels  would  he  expressed  to  .a  hundredth.  The 
decision  with  respect  to  the  absolute  size  of  a  difference  which  will  be 
accepted  as  significant  therefore  must  be  approavhed  with  can!  ion. 
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Overview 


Despite  the  plethora  of  proponents  and  techniques  available  for  tech- 
nological  forecasting,  the  majority  view  of  those  who  have  tried  to  assess 
this  discipline  seems  to  be  something  less  than  optimistic.  Dory  and  Lord 
(1970)  in  an  article  asking  the  question,  "Does  Technological  Forecasting 
Really  Work?  ,  concluded  that,  "Technological  forecasting,  the  perception 
and  interpretation  of  change,  appears  to  represent  a  far  more  difficult 
problem  than  is  currently  indicated  by  its  proponents  or  addressed  by  their 
techniques."  Another  criticism  was  voiced  by  Bright  (1970),  "It  (techno¬ 
logical  forecasting)  remains  Inadequate  for  some  of  management's  needs 
because:  (a)  ;  (b)  forecasts  of  technical  capabilities  do  not  necessarily 

deal  with  their  diffusion,  input  requirements,  and  impact." 

In  a  broader  context,  an  Interagency  Task  Group  on  Technological  Fore¬ 
casting  in  the  Federal  Government  (1966)  concluded,  "There  Is  little,  if  any, 
evidence  of  successful  attempts  to  forecast  actual  5  to  10  year  requirements 
for  skill  in  broad  classes  of  occupations."  In  this  same  time  frame,  Hacke 
(1967)  (see  Appendix  A-3J,  discussing  the  feasibility  of  anticipating 
economic  and  social  consequences  of  technological  innovation,  talked  about 
two  aspects  of  forecasting,  parameters  and  patterns.  According  to  this 
author,  the  forecasting  of  parameters  has  been  the  subject  of  mucli  work.  As 
Hicke  states,  "The  following  examples  demonstrate  that  there  is  no  lack  of 
methods  for  forecasting  trends  in  socio-economic  parameters:...."  Hacke 
continues,  "in  the  projection  of  economic  and  social  patterns,  by  contrast, 
little  has  been  done  that  goes  beyond  historical  analogy  and  expert  opinion." 
Earlier  Schoeffler  (1955)  was  more  critical  in  his  evaluation  of  explanations 
of  changes  in  patterns  by  categorizing  them  as  "story  telling."  Though  this 
statement  was  made  almost  20  years  ago,  the  situation  apparently  has  not 
been  completely  reversed. 

A  summarization  of  the  criticism  of  technological  forecasting  methods 
was  succinctly  made  by  Lenz  (1969)  in  the  following  quotes. 

In  practice  it  may  often  be  observed  that  data  for 
parameters  of  performance  which  would  be  useful  for 
measurement  and  prediction  of  progress  are  very 
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difficult  to  obtain.  Performance  is  sometimes  not 
measured  at  all,  or  Is  often  not  measured  at  regular 
Intervals  .  .  ..Tfius  without  a  cle.ir  and  specific 
knowledge  of  exai’t  limits  of  present  and  past  per¬ 
formance,  It  is  not  unusual  to  find  a  reluctance 
toward  projecting  or  publishing  forecasts  of  future 
performance  in  the  same  area. 

Lenz  continues,  "The  development  of  a  useful  quantitative  forecast,  using 
even  the  simplest  methods  of  trend  forecasting,  is  likely  to  become  a 
lengthy  research  project."  Lenz  summarizes  the  products  of  technological 
forecasting  by  stating,  "The  scarcity  of  good  examples  (good  forecasts)  Is 
compounded  by  the  difficluty  of  the  search  through  large  quantities  of 
material  labeled  as  forecasts  of  future  trends  to  find  the  few  which  are 
useful."  (Support  for  this  statement  may  be  found  by  consulting  the  bibli¬ 
ography  compiled  by  Harrison  (1971).  The  author  identifies  more  than  800 
sources  pertaining  to  technological  forecasting  spanning  the  years  1936  to 
1970.) 

Roberts  (1969)  (see  Appendix  A-4),  critically  appraising  exploratory 
forecasting,  concluded  that  most  of  the  existing  methods  are  variants  of 
simple  trend  extrapolation  procedures  and  have  limited  utility  In  the  current 
dynamic  technological  environment. 

This  objection  to  the  lack  of  development  of  more  than  a  simplistic 
approach  method  is  also  made  by  Jantsch  (1967)  in  his  statement  that  only  a 
limited  number  of  influencing  factors  are  taken  into  account  and  mathematical 
formulations  "do  not  yet  include  even  all  of  these  recognized  factors." 

Of  the  Delphi  procedure,  Roberts  speculates  about  "whether  a  Delphic 
sampling  of  appropriate  whiz  kids  would  have  predicted  an  effective  inter¬ 
continental  ballistic  missile  when  Vannevar  Sush  failed  in  his  prophecy!" 

Roberts  argues  strongly  for  the  need  of  a  "learning  model"  which  would 
combine  a  real-time  system,  on  which  data  could  be  collected  and  interpreted, 
with  a  forecasting  model.  This  combination  could  allow  "parameter  and 
possibly  even  structural  changes  in  the  forecasting  model  based  on  experienced 
model  ...  performance." 

Normat i  sting  is  viewed  by  Roberts  as  failing  le  lirect  a  suf¬ 
ficient  a  toward  verification  of  the  correctness  of  the 


assumed  relationships  between  allocated  resources  and  generated  technology. 
Supporting  this  statement  is  the  Dory  and  Lord  (1970)  comment,  "We  agree  with 
the  literature  that  normative  forecasting  is  a  better  answer  than  the  explora¬ 
tory  approach  for  many  applications.  We  are  not  sure  either'  attacks  the  real 
need  for  information." 

Roberts  concludes  his  appraisal  by  calling  for  a  technique  integrating 
exploratory  and  normative  forecasting  methods  and  making  use  of  feedback  loops. 

In  retrospect  three  points  need  to  be  considered.  First,  although  there 
are  a  considerable  number  of  techniques  currently  being  used  to  forecast  tech¬ 
nology,  none  of  these  are  singularly  more  effective.  As  in  any  discipline 
they  should  be  viewed  as  basic  tools,  each  with  its  own  peculiar  unique  ad¬ 
vantages  and  disadvantages.  To  a  degree  the  selection  of  a  given  technique 
is  predetermined  by  the  purpose  of  the  research.  In  addition,  a  consideration 
must  be  made  as  to  the  type  of  information  that  is  desired  as  a  final  outcome. 
Drawing  upon  Metroff  and  Turoff  (1973),  an  adaptation  of  their  conceptualiza¬ 
tion  of  information  types  is  provided  in  Table  1.  As  can  be  seen,  the  type  of 
consideration  ranges  from  the  relatively  simple  one  of  establishing  the  tech¬ 
nical  feasibility  of  a  specific  short  term  goal  such  as  the  use  of  Wankcl 
engines  in  aircraft  to  the  more  complex  long  term  goal  exemnliflcd  by  the  re¬ 
duction  of  population  growth  over  the  next  90  years.  The  selection  of  a 
specific  methodology  may  be  strongly  influenced  by  the  type  of  outcome  informa¬ 
tion  expected.  For  example,  type  IV  information,  such  as  how  to  geographically 
allocate  Air  Force  resources  to  maximize  Air  Force  effectiveness  in  meeting 
national  objectives,  /ould  probably  requite  some  type  of  matrix  consideration 
in  the  attempted  forecast. 

Second,  the  technique  selected  only  provides  a  methodology  not  a  certainty. 
The  technique  can  only  be  asked  to  be  effective  If  the  data  is  in  sor.e  way  re¬ 
liable,  valid  and  credible.  The  strongest  technique  cannot  withstand  the  vib¬ 
ration  of  inconsequential  data.  Therefore,  a  far  greater  effort  must  be  pi  u 
upon  data  measurement  and  operational  definitions  of  variables  being  consi 
Prior  to  this  time  most  technological  forecasting  and  assessment  has  been 
on  either  highly  restrictive  quantitative  data;  l.e.,  weight  versus  thriis 
engine  performance,  which  mav  not  reflect  the  conceptual  variable  of  inter 

performance  (Alexander  and  Nelson,  1973),  or  on  extremely  general  observation 
which  lack  the  reliability  desired  in  measuring  a  variable  (Slmmonds,  1973). 
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Table  1.  Types  of  Information  Obtained  in 

'technological  Forecasts  and  Assessments 

(Adapted  From  Mitroff  &  Turoff  1973) 


j  TYPE 

DESCRIi'TION 

1 

j  I  Feasible  Technological 

1  Developments 

i 

Feasible  usually  means,  in  this  context, 
technically  feasible  if  the  "required" 
resources  are  Invested  or  available. 

!  II  Potential  Applications 

This  is  any  possible  application  of  the 
previous  technological  developments  with¬ 
out  regard  necessarily  to  values  -  i.e., 
whether  the  application  is  good  or  bad. 

III  Significant  Applications 

i 

This  is  some  subset  of  "all"  potential 
applications  or  a  transformation  to  some 
set  of  significance  tc  the  intent  of  the 
study. 

!  IV  Potential  Consequences 

1 

i 

Any  consequences,  good  or  bad,  which  may 
affect  opinions  of  scenarios  about  the 
future,  or  our  interpretation  of  the  past.j 

V  Policy  or  Resource 

Allocation  Issues 

1 

Concerned  with  the  decision  issues  under 
examination  or  arising  as  a  result  of  po¬ 
tential  consequences  of  decisions. 

VI  Potential  Resolution 
of  Issues 

The  controls  that  can  be  imposed  to  effect 
the  likelihood  of  various  developments, 
applications,  and  consequences. 

20 


Third,  what  we  are  attempting  to  forecast  Is  In  Itself  not  generally  de¬ 
fined.  That  Is,  technology  is  an  extremely  broadly  Interpreted  word  which  can 
be  anything  from  a  broad  scientific  discovery,  such  as  fusion,  to  a  highly 
specific  item,  jet  engines.  In  either  case  the  research  scientist  is  inter¬ 
ested  in  determining  the  Impact  of  some  perhaps  almost  unknown  event  in  terms 
of  a  very  concrete  system  or  outcome.  The  very  fact  that  someone  may  be  con¬ 
cerned  with  the  Impact  of  the  laser  upon  photography  does  not  in  Itself  estab¬ 
lish  any  given  approach.  The  technological  Innovation  must  be  defined  in  some 
specific  set  of  characteristics  which  in  turn  must  be  related  to  the  photo¬ 
graphic  characteristics  of  interest.  The  vast  latitude  between  the  specificity 
of  forecasts  creates  one  of  the  major  hazards  ir  any  logical  synthesis  of  avail¬ 
able  research.  This,  perhaps,  suggests  that  in  order  to  study  the  question,  a 
series  of  Interrelated  studies  may  be  necessary  to  establish  a  conceptual  net¬ 
work  of  data  which  will  be  the  result  of  several  different  approaches  to  the 
problem. 

METHODS  FOR  RELATING  STATES  OF  TECHNOLOGY  TO  THF  IMPACT  ON  HUMAN  RESOURCES 

Qualitative  and  Quantitative  Personnel  Requirements  Information 

Through  a  process  of  evolution,  the  Air  Force  developed  a  concept  for 
stating  personnel  and  manning  requirements  of  developing  systems.  This  concept 
has  undergone  a  series  of  modifications  and  name  changes;  it  is  currently  known 
as  Qualitative  and  Quantitative  Personnel  Requirements  Information  (QQPRI) 

(USAF,  1961).  The  QQPRI  is  a  statement  of  projected  manpower  required  to 
operate  and  to  maintain  a  new  weapon  system.  It  is  usually  based  on  a  task 
analysis  and  is  summarized  in  terms  of  an  "average"  operational  unit.  The 
summary  lists  the  numbers  and  career  areas  of  personnel  required.  It  also 
provides  an  indication  of  the  skill  level  of  each  individual.  Folley,  Fairman 
and  Jones  (1960)  conducted  a  survey  of  the  literature  on  methods  for  prediction 
of  Air  Force  personnel  requirements.  The  overwhelming  majority  of  the  121 
documents  abstracted  dealt  with  QQPRI  and  its  antecedents.  Only  1  of  the  121 
reports  surveyed  addressed  the  question  of  elaboration  of  a  procedure  for 
estimating  manpower  requirements  (Ray  et  al .  1957).  This  procedure  used  a 
rating  scheme  developed  by  Ghiselli  and  Brown  (1955)  which  classified  skills 
into  3  levels  of  complexity.  The  approach  was  based  on  use  of  the  same  inputs 
involved  in  QQPRI  generation  and  appeared  to  result  in  the  same  product,  speci¬ 
fication  of  QQPRI  data.  Thus,  the  Ray  procedure  presented  no  basic  improvement 
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over  the  QQPRI  method.  The  reader  Interested  in  more  detail  concerning  QQPRI 
is  referred  to  Losee  et  al  (1961)  and  a  later  effort,  Demaree  et  al  (1962), 
which  detailed  the  approach  methodologies  used  to  produce  QQPRi  and  made  dis¬ 
tinctions  between  QQPRI  and  other  products  in  the  Personnel  Subsystem  effort. 
Both  documents  were  Intended  to  serve  as  guides  to  the  preparation  of  QQPRI  for 
use  in  weapon  system  development  efforts. 

A  look  at  extra  Air  Force  attempts  to  establish  procedures  for  specify¬ 
ing  personnel  and  training  requirements  for  incoming  weapons  systems  is  pro¬ 
vided  by  Rupe  (1963).  Haines  and  Gael  (1963)  (see  Appendix  A-5)  conducted  a 
survey  of  manning  estimation  techniques  used  in  defense  industries.  In  all 
cases  surveyed,  QQPRI  was  used  as  rhe  approach  method.  The  resultant  QQPRI 
products  were  found  to  be  based  on  five  methods  of  obtaining  inpu,"^  data. 

These  were;  expert  estimate,  historical  'omparison,  task  analysis,  "sovereign 
factors"  (e.g.,  cost  used  a.s  an  index  of  complexity  together  with  a  queuing 
theory  approach),  and  modeling  approaches.  Kssentially,  the  QQPRI  concent 
has  not  changed  since  these  earlv  reviews. 

The  search  for  a  method  to  specily  system  manning  requirements  earlier 
in  the  development  cycle  of  the  system  has  led  to  investigation  and  experi¬ 
mental  application  of  simulation  techniques  with  AFM  66-1  (1972)  type  mainte¬ 
nance  data  along  with  other  Inputs  as  the  information  base.  These  procedures 
to  date  are  not  documented  in  published  material.  '' 

Simulation  Techniques 

The  feasibility  of  specifying  manning  requirements  of  advanced  systems 
by  means  of  mathematical  modeling  approaches  has  been  investigated  for  at 
least  a  decade.  Barton  et  al  (1964)  (see  Appendix  A-6)  used  two  mathemati¬ 
cal  techniques  (queuing  and  a  linear  programming  model)  to  compute  manning 
requirements  and  training  needs  in  a  developing  system.  The  authors  pointed 
out  that,  in  practice,  failed  systems  or  units  pile  up  in  lines  waiting  for 
or  men  are  incompletely  utilized.  '  They  felt  this  model  adequately 
dealt  with  these  conditions.  A  set  of  queuing  tables  for  use  with  this 
model  was  generated  by  Purvis,  McLaughlin  and  Mallory  (1964).  Further 

t  Personal  communication  with  R.J.  Schiffler,  (USAF,ASD).  The  interested 
reader  is  referred  to  AFHRL  prepublication  draft:  York,  Michael  L.  (Major, 
USAF),  A-IQ  Manpower  Requirements  Development  Test,  14  September  1973. 
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investigation  of  this  application  of  queuing  techniques  was  made  by  Purvis, 
Mallory  and  McLaughlin  (1965)  (see  Appendix  A-7)  with  the  Intent  of  estab¬ 
lishing  the  validity  and  reliability  of  the  procedure  for  predicting  manning 
requirements  for  weapons  systems.  The  approach  used  by  the  authors  was  the 
application  of  the  procedure  to  an  operational  system  (the  F-105D  fire  con¬ 
trol  system)  and  to  a  system  in  development  (the  C-141) .  The  conclusion 
reached  was  that  the  approach  was  usable  and  that  "the  technique  clearly 
establishes  the  validity  of  the  finite  cyclical  queuing  process  as  an  accu¬ 
rate  description  of  the  F-105  fire  control  system 

Snyder  and  Askren  (1968)  (see  Appendix  A-8)  support  the  statements  of 
Purvis,  Mallory  and  McLaughlin  with  their  conclusion,  "queuing  techniques 
are  valid  and  reliable  for  predicting  requirements  and  making  trade-offs 
between  manning  and  personnel  skills 

Approaches  taken  by  other  investigators  to  simulation  model  solutions 
for  specifying  the  impact  of  technological  innovations  on  human  resource 
parameters  have  included  Bradley  (1968)  and  Floyd  (1968).  Bradley  suggested 
a  Monte  Carlo  simulation  model  emphasizing  prediction  of  manpower  and  spares 
requirements.  This  model  was  computer  based,  but  provisions  were  made  for 
manual  application.  A  different  modeling  approach  was  advanced  by  Floyd,  who 
suggested  a  mathematical  model  leading  to  the  development  of  trend  curves 
for  prediction  when  two  conditions  were  satisfied;  first,  an  ultimate  limit 
could  be  identified  and,  second,  when  at  least  two  data  points  were  available. 
Current  efforts  under  Air  Force  direction  include  the  experimental  effort 
mentioned  in  the  section  above,  which  is  an  approach  entailing  mathematical 
modeling  of  future  aircraft  systems'  human  resources  requirements  based  on 
AFM  66-1  maintenance  data  assembled  on  a  current  comparable-mission  aircraft. 
This  effort  resulted  in  the  development  of  a  computerized  model  which  con¬ 
siders  operations  scenarios  and  task  data  to  project  resource  levels  of  men, 
spares  and  aerospace  ground  equipment  (AGE).  The  research  has  been  completed 
and  a  series  of  reports  are  being  drafted.  The  technique  has  been  adapted 
for  some  current  Air  Force  systems. 

Statistical  Approaches 

Closely  related  to  modeling  techniques  are  statistical  approaches  to  the 
prediction  of  the  impact  of  technological  innovation  on  human  resources.  The 
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statistical  approaches  to  this  problem  have,  to  a  large  extent,  emphasized 
regression  analysis  procedures  to  arrive  at  an  estimate  of  the  effects  of 
equipment  design  characteristics  on  skill  requirements  or  job  performance. 

Meister,  Finley  and  Thompson  (1971)  used  a  multiple  regression  analysis 
as  one  of  the  steps  in  a  process  which  they  termed  "a  sort  of  ’poor  man's 
factor  analysis'."  For  a  summary  of  the*  procedure  advanced  by  these  authors, 
refer  to  the  expanded  summary  included  in  Appendix  ‘-9. 

The  authors  identified  four  factors  as  prime  determinants  of  maintenance 
performance.  These  were  accessibility,  amount  of  diagnostic  information, 
equipment  configuration,  and  technician  capability. 

The  expected  recommendation  for  replication  with  an  expanded  sample  size 
to  increase  conf  idence  in  results  was  made.  An  interesting  conclusion  enumer¬ 
ated,  but  not  emphasized,  by  the  authors  was  the  suggestion  that  length  of 
experience  alter  skill  level  qualification  was  not  an  important  design  param¬ 
eter.  In  view  of  the  potential  training  and  design  consideration  impacts 
represented  by  such  a  condition,  anv  replication  of  this  study  should  direct 
specific  attention  to  collection  of  data  addressing  this  issue. 

Smith,  Blanchard  and  Westland  (1971),  investigated  subjective  techniques 
for  generating  human  performance  repair  time  data  required  for  predictions  of 
maintainability  of  proposed  equipment.  Data  generated  was  compared  against 
AFM  f)6-l  maintenance  data  by  means  of  regression  analysis  techniques.  The 
relationship  was  found  to  be  linear  (v  =  l.2756x  +  0.7021)  with  a  positive 
correlation  of  0.86.  The  authors  concluded  that  subjective  judgment  could  he 
usi‘d  to  predict  maintainability  of  future  systems. 

The  relationship  between  equipment  design  characteristics,  training  costs 
and  difficulty,  and  job  performance  was  Investigated  by  Lintz,  Loy,  Hopper 
and  Potempa  (1973)  (see  Appendix  A-10).  Thc^  approach  was  through  the  use  of 
a  principal  components  (stepwise  regression)  analysis  to  derive  prediction 
equations  for  performanc'e  time,  errors,  training  time,  and  training  equipment 
cost.  Variables  correlating  significantly  with  performance  were  subjected  to 
a  factor  analysis.  This  procedure  resulted  in  the  identification  of  six 

factors.  I’he  .authors  concluded  that  tlie  procedures  followed  provided  a 
quantitative  method  for  predicting  tiu*  variable's  undi*r  study  .and  that  such 


24 


! 

f 


predictions  could  be  used  as  considerations  in  a  system  design  process.  The 
authors  call  for  validation  of  the  procedure  through  following  a  system  design 
effort  from  conceptualization  to  operational  use. 

A  subsequent  study  employing  the  same  methodology  was  conducted  by  Lintz, 
Loy,  Brock,  and  Potempa  (1973)  (see  Appendix  A-11) .  Organizational  and 
intermediate  level  maintenance  data  were  collected  on  10  avionics  sub'?ystems. 
Within  these  subsystems,  selected  equipment  design  characteristics  were  either 
measured  or  subjectively  rated.  In  addition,  16  personnel  variables  were 
used.  Maintenance  tasks  of  differing  levels  of  difficulty  were  used  for 
estimating  performance  times  and  error  probabilities  for  both  high  and  low 
skill  level  personnel. 

Design,  personnel,  performance,  anddlfflculty  variables  were  Intercor- 
related  and  were  entered  into  regression  and  factor  analyses.  Equations 
were  derived  for  predicting  maintenance  task  difficulty  and  maintenance  per¬ 
formance  times.  Maintenance  performance  time  was  predicted  through  a  con¬ 
sideration  of  personnel  and  task  difficulty  variables. 


Multiple  correlation  coefficients  (Rs)  for  predicting  times  from  design 
characteristics  ranged  from  0.87  to  0.93.  Task  difficulty  coefficients 
ranged  from  0.73  to  0.80.  Factor  analyses  on  performance,  personnel  and 
design  variables  yielded  some  17  factors  over  the  different  maintenance 
levels  studied.  The  authors  concluded  that  the  results  supported  the  merit 
of  such  an  approach  to  predicting  maintenance  performance  times  and  task 
difficulty  and  felt  an  extension  of  the  procedure  to  additional  avionics 
equipment  was  in  order. 


A  factor  analysis  approach  was  also  used  earlier  by  Topmiller  (1964)  as 
an  approach  method  for  conducting  human  engineering  analyses  and  predicting 
system  maintainability.  A  more  global  approach  was  employed  in  that  the 
author  was  primarily  concerned  with  equipment  classes  rather  than  directing 
attention  to  the  LRU  level  -  the  object  of  study  in  Lintz,  Loy,  Brock  and 
Potempa  (1973).  Topmiller  reported  that  Increased  predictive  efficiency  was 
experienced  with  his  approach  when  equipment  was  classified  into  homogeneous 
sets. 
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Cross  I’rojt-ction  Molhocis 

Most  frequently  encountered  i;i  reviewing  the  literature  are  studies  of 
the  elfeets  of  t eehnol og lea  1  change  on  skill  level  where  skill  level  is 
treated  only  in  terms  of  gross  projections. 

Crossman  (1960)  provided  a  scheme  for  study  of  types  of  automation  which 
involved  classifying  the  types  into  continuous  flow  production,  programmed 
machine,  and  centralized  remote  control  activities.  One  of  the  impacts  of 
automation,  according  to  Crossman,  would  be  the  developing  requirement  for  a 
mu  1 1 i sk i 1  led  maintenance  man,  a  skill  type  he  alluded  to  as  a  "polyvalent 
crattsman."  Davis  (l'‘a2)  concluded  that  levels  required  for  semiskilled  mass 
production  jobs  would  be  suitable  for  operators  of  automated  equipment. 
Beaumont  anci  Helfgoit  (1964)  reporting  on  a  study  of  36  companies  concluded 
that  worker  skill  requirements  were  fundamentally  unchanged  by  technological 
innovation.  Bright  (1966)  arrived  at  the  disparate  conclusion  that  automated 
machinerv  tended  to  require  less  operator  skill.  Crossman  and  Laner  (1969) 
(see  Appendix  A-12)  separated  skills  into  direct  and  indirect  labor  and  re¬ 
ported  finding  .in  increase  in  direct  labor  skill  requirements  but  concluded 
Chat  with  the  inclusion  of  indirect  labor  into  the  consideration  there  was  no 
overall  increase  in  rie.an  skill  levels  required  as  a  result  of  technological 
change. 

Kraft  (1970)  called  for  study  of  specific  skill  responses  required  at 
various  stages  of  automation.  He  felt  that  such  an  approach  might  provide  the 
information  needed  for  adequ.ite  manpower  planning. 

Relevance  Tree  Approaches 


The  earlier  referenced  efforts  of  Wells  (1958)  apparently  represented 
the  first  application  of  a  relevance  tree  approach  to  the  problem  of  the  tech¬ 
nological  forecasting  for  impact  considerations.  Continued  attention  was 
given  to  this  procedure  by  Esch  and  Jest  ice  (Kscb,  1965)  with  the  result  that 
"relevance  numbers"  were  attached  to  the  branches  of  the  tree  to  permit  se¬ 
lect  ii)n  of  an  optimized  path  through  the  tree. 

Hwang  (1970)  was  one  ol  the  early  Investigators  to  perceive  the  relevance 
tree  approacli  as  a  vehicle  for  study  of  specific  weapon  system  prediction 
(|uestions.  Ilwaii)',  at  I  einpl  ed  to  csl.iblisii  the  comparability  ol  risk  an.ilvsis 
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to  a  systems  analysis  approach.  He  used  a  decision  tret*  approach  in  consort 
with  methods  of  risk  analysis  to  predict  risk  in  materials  acquisition  for 
the  U.S.  Army. 

In  contradiction  to  a  conclusion  reached  by  Meister  and  Farr  (1967), 
Llntz,  Askren,  and  Lott  (1971)  reported  that  design  engineers  did  consider 
human  resources  data  if  it  was  available  and  that  when  considered,  this  data 
did  affect  resulting  designs.  The  authors  recommended  that  human  resources 
data  be  Included  as  one  of  the  design  parameters  in  weapon  system  development. 
In  seeking  an  answer  to  the  problem  of  quantitative  specification  of  human 
resources  Impact  data  in  advance  of  the  actual  application  of  a  technology, 
one  of  the  above  authors,  Askren,  in  collaboration  with  Korkan  (1971)  (see 
Appendix  A-13)  investigated  the  feasibility  of  the  use  of  a  relevance  tree 
approach  as  a  vehicle  for  relating  human  resources  data  to  design  decision 
alternatives.  The  relevance  tree  developed  by  the  authors  reflected  weapon 
system  decision  options  and  was  termed  a  "Design  Option  Decision  Tree  (DODT)." 

The  underlying  basis  for  jthe  development  of  this  particular  research 
approach  apparently  lies  in  th !  fact  that  in  the  design  of  a  weapon  system, 
the  system  designer  can  rely  oi  a  store  of  past  engineering  data  which  can  be 
considered  in  selecting  the  hardware  design  option  to  follow.  However,  as 
these  authors  succinctly  indicate. 

In  the  case  of  human  resources  data,  there  is  no  such 
bank  of  historical  data.  Thus,  the  need  exists  for 
the  generation  cf  data  descriptive  of  Air  Force  human 
resources  implications  of  the  various  design  options 
identified  in  the  decision  trees.  The  generation  of 
these  data  likely  will  require  a  combination  of  efforts, 
such  as  collection  of  historical  and  field  data,  and  the 
generation  of  data  through  simulation  and  psychophysical 
procedures. 

The  authors  concluded  that  design  option  decision  trees  appeared  to  provide  a 
vehicle  for  permitting  a  systematic  consideration  of  human  resources  factors 
involved  in  a  specific  design  decision. 

In  a  follow-on  study,  Askren,  Korkan  and  Watts  (1973)  (see  Anpendix  A-14) 
Investigated  the  feasibility  of  measuring  the  sensitivity  of  human  resource 
parametirs  data  to  design  trade-off  problems  reflected  in  a  DODT.  Eight 
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trado-off  problems  were  chosen  from  an  aircraft  propulsion  tree  and  psycho¬ 
metric  data  collected  concerning  the  effect  of  these  design  options  on 
selected  human  resources  topics.  The  authors  concluded  that  the  study  demon¬ 
strated  the  feasibility  of  the  use  of  psychometric  methods  in  conjunction  with 
a  DODT  to  develop  selected  human  resources  data  for  use  in  system  design 
decisions. 

Other  Approaches 

The  attempts  to  relate  the  effects  of  technological  innovation  on  human 
resource  variables  over  the  rec.  t  past  appear  to  have  emphasized  rather  re¬ 
stricted  avenues  of  investigation.  The  approaches  represented  by  QQPRT  gen¬ 
eration,  regression  analysis,  queuing  and  relevance  tree  techniques  dominate 
the  relatively  sparce  literature  in  this  area.  Isolated  investigations  of 
the  feasibility  of  application  of  other  approaches  are  found,  but  generally 
are  not  the  subject  of  more  than  one  investigation.  Thus,  in  1961,  Thomas 
et  al  (see  Appendix  A-15)  described  a  method  they  had  developed  for  generating 
data  on  the  effects  of  various  levels  of  automation  on  human  performance  in  a 
man-machine  system.  The  method  was  advanced  as  a  means  of  studying  system 
control  problems  anticipated  in  the  design  of  an  ad\anced  system. 

Grossman  (1965a)  reported  on  an  approach  Intended  to  define  skill  level 
on  a  scale  independent  of  the  content  of  the  skill.  The  approach  suggested  en¬ 
tailed  measuring  the  man-hours  expended  in  developing  the  skill  resulting  in  a 
scale  which  would  be  "divisible  into  some  six  or  seven  levels."  The  Crossman 
scale  basically  would  represent  a  parallel  system  to  the  Air  Force  scheme  of 
skill  levels  and  would  be  developed  on  a  less  satisfactory  data  base. 

In  1969  (Melster  et  al),  an  investigation  was  conducted  concerning  the 
design  engineer  s  concept  of  the  relationship  between  design  characteristics 
and  technician  skill  level.  The  authors  asked  their  sample  of  design  engineers 
to  rank  ten  statements  of  "different  levels  of  maintenance  skill"  along  a 
ten-point  scale.  They  concluded,  "Engineers  claim  to  be  capable  of  differ¬ 
entiating  ten  levels  of  personnel  skill." 

Dean  (1959)  in  an  unpublished  dissertation,  reported  on  an  Investigation 
of  the  primary  intellectual,  perceptual,  visual,  interpersonal,  procedural 
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and  muscular  skill  requirements"  Imposed  by  the  introduction  of  automated 
equipment  in  seven  case  studies  of  industry.  A  Job  Description  Check  List, 
developed  by  Storey  (1958),  was  used  to  provide  quantitative  indices  of  the 
skill«components  listed  above  for  each  case  studied,  both  before  and  after 
automating.  Dean  found  no  effects  of  automation  in  75  per  cent  of  the  obser¬ 
vations.  In  the  remainder  of  the  observations,  the  number  of  skill  increases 
were  approximately  equal  to  the  number  of  decreases. 

Schulze  (1971)  reported  on  an  attempt  to  relate  equipment  design  features 
to  selected  psychophysical  indices.  Though  finding  no  statistically  signifi¬ 
cant  differences,  the  author  reported  strong  trends  for  a  majority  of  the  in¬ 
dices  to  be  higher  during  the  more  difficult  tasks. 

From  the  information  available  it  appears  that  the  relationship  between 
existing  states  of  weapon  systems  and  human  resource  parameters  has  only  been 
addressed  in  a  crude  and  unsystematic  manner.  The  prognostic  needs  of  the 
armed  services  forged  the  concept  of  QQPRI,  but  this  concept  has  lacked  a 
specialized  technique  which  would  provide  adequate  information.  Some  of  the 
methods  used  establish  acceptable  projections  of  manpower  but  are  usually 
applied  after  the  weapon  system  has  transitioned  into  the  production  phase. 
Initial  efforts  have  shown  that  human  resource  parameters  will  affect  weapon 
system  design  if  the  Impact  of  a  given  trade-off  has  reliable  data  provided. 

There  is  presently  no  body  of  information  that  would  provide  the  required 
relationships  betweer  design  characteristics  and  human  resource  parameters. 

The  design  option  decision  tree  technique  provides  a  network  instruction  tech¬ 
nique  that  allows  us  to  reduce  a  system  problem  into  a  series  of  functional 
decision  points.  This  certainly  provides  Insight  into  complex  system  prob¬ 
lems,  but  it  does  not  establish  the  necessary  relationships.  It  is  concluded 
that  the  methodology  currently  available  in  the  area  of  predicting  human  re¬ 
source  requirements  for  a  given  system  is  not  any  more  advanced  than  fore¬ 
casting  methodology,  and  it  deals  almost  exclusively  with  the  three  manpower 
considerations  of  quantity,  job  type  and  skill  level. 

Overview 

Research  attempting  to  relate  the  Impact  of  technological  innovations  on 
human  resources  has,  to  a  large  extent,  emphasized  the  investigation  of  the 
effect  of  automation  on  overall  manpower  skill  requirements.  The  more  com- 
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plfx  issui‘  (if  antic  ipating  the  specific  changes  which  application  of  a  tech- 
nuloKiial  advance  generates  within  a  particular  skill  classification  has  been 
neither  adequately  treated  nor  resolved. 

An  extensive  review  of  earlier  efforts  directed  toward  assessment  of  the 
impact  of  automation  on  manpower  skill  requirements  was  conducted  by  the 
fntcTnal ional  Labour  Office  (1965).  This  effort  reviewed  160  studies,  49  of 
wliiili  wi  rc‘  at  comp !  ished  in  the  United  States.  Erch  of  the  studies  was  cate¬ 
gorized  in  terms  of  the  kinds  of  technological  change  involved  and  was  class- 
ificni  with  respect  to  its  effect  on  selected  social,  economic,  occupational, 
and  personal  factors.  The  studies  in  this  review,  and  many  of  the  more  re¬ 
cent  studies,  have  dealt  with  skill  requirements  in  terms  of  gross  census- 
type  predictions. 

In  criticism  of  earlier  work  in  this  area  Crossman  (1965b)  concluded, 

"To  this  date,  we  have  essentially  no  valid  models  which  will  predict  the 
rate  o!  progress  of  automation  in  any  single  case,  nor  the  likely  changes  in 
skill  input ." 

That  attention  is  still  directed  at  the  level  of  gross  projection  is  re¬ 
flected  in  sources  such  as  Buechner  (1972)  (see  Appendix  A-16),  who  con¬ 
cluded  that  technological  change  and  associated  automation  served  to  reduce 
demands  on  labor,  but  cautioned  that  the  nature  of  his  data  did  not  allow 
consideration  of  individual  jobs. 

DeGreeno  (1974?)  asserts  that  in  spite  of  20  years  study,  a  categorical 
statement  concerning  the  effect  of  automation  on  skill  requirements  cannot 
be  made. 

A  major  criticism  leveled  against  QQPRI  has  been  the  relatively  late 
stage  in  the  system  development  cycle  at  which  personnel  information  became 
available.  Attempts  to  develop  specific  human  resource  information  at  earliei 
points  in  the  system  development  cycle  have  led  to  Investigations  pursuing 
the  approaches  summarized  in  the  sections  above.  To  date,  no  approach  has 
provided  the  answers  sought.  At  least  one  individual  (DeGreene,  1974?)  ad¬ 
dressing  this  area  of  concern,  pessimistically  concludes,  "...  it  will  prob¬ 
ably  be  at  least  years  before  a  predictive  or  even  an  explanatory  theory  of 
technological  changes  (as  related  to  manpower  resources)  is  developed." 
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DeGreene  calls  for  development  of  a  systems  theory  integrating  the  various 
approaches  of  psychology,  engineering,  computer  sciences,  economics,  and 
other  related  fields  to  deal  with  this  problem. 

POTENTIAL  METHODS  FOR  PREDICTION  OF  IMPACT  OF  TECHNOLOGICAL  INNOVATIONS  ON 
HUMAN  RESOURCE  PARAMETERS 

Each  (1965)  reported  on  a  modification  of  the  PATTERN  relevance  tree 
undertaken  by  the  Honeywell  Corporation  in  an  attempt  to  identify  present  aiid 
future  needs  in  bio-medicine.  This  effort  was  referred  to  as  Project  Medi¬ 
cine."  The  term  "medicine"  in  the  title  was  an  acronym  standing  for  medical 
instrumentation  and  control  identified  and  numerically  evaluated.  This  pro¬ 
cedure  used  the  decision  structuring  and  relevance  number  assignment  tech¬ 
niques  developed  for  PATTERN  but  redefined  the  levels  within  the  tree.  For 
example,  in  the  hardware  trees,  the  primary  activity  level  is  the  task  level. 
In  the  bio-medical  tree,  this  level  was  concerned  with  such  activities  as  diag¬ 
nosis,  palliative  procedures,  curative  procedures,  and  prevention.  System 
and  subsystem  levels  of  the  hardware  oriented  tree  became  major  body  activ¬ 
ities  (e.g.,  blood  circulation)  and  body  organs,  respectively.  In  this 
fashion,  the  relevance  tree  was  employed  by  Honeywell  to  plan  a  research  and 
development  program  in  medical  electronics,  thus  providing  a  demonstration  of 
the  generallzability  of  the  relevance  tree  approach. 

Investigating  the  relationship  between  advanced  technology  and  work  group 
behavior  in  a  replication  study,  Taylor  (1970)  reported  support  tor  earlier 
conclusions  that  a  pattern  of  greater  peer  leadership  was  more  evident  in 
groups  exposed  to  more  advanced  technology  and  that  an  advanced  technology 
climate  facilitated  planned  social  change  efforts. 

Bezdek  and  Getzel  (1973),  using  gross  projection  methods,  nredicLed  the 
skill  requirements  for  three  future  socio-economic  environments:  a  status 
quo,  a  social-welfare,  and  a  defense  economy.  They  concluded  that  fo--  all 
three  futures  manpower  requirements  at  upper  skill  levels  would  be  sub¬ 
stantially  increased.  The  choice  of  national  priorities  was  viewed  by  the 
authors  as  exercising  a  significant  effect  on  sVill  types  within  job  famil¬ 
ies.  This  factor  was  viewed  as  a  compounding  problem  making  accurate  pre¬ 
diction  of  education,  vocational  and  skill  requirements  very  difficult. 
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Tlie  application  of  the  material  in  this  and  in  many  of  the  other  arti¬ 
cles  in  this  area  is  restricted  by  the  same  problem  encountered  earlier; 
most  articles  address  themselves  to  trend  extrapolations  and  gross  predic¬ 
tions  (e.g.,  Gordon  and  Ament,  1969),  thus  provide  little  useful  insight  for 
answer  to  questions  concerning  specific  Impact  effects  on  human  resources. 

This  sentiment  is  supported  by  Hacke  (1967)  who  stated,  "It  is  not  feasible, 
however,  to  forecast  quantitatively  the  trends  in  a  (social  or  economic)  par¬ 
ameter  or  pattern  with  the  certainty  that  the  oiitcouie  will  fall  within  the 
stated  confidence  limits 

i)uANTIFlCAT10N  OF  HUMAN  PERFORMANCE  FACTORS  IN  SYSTQl  DEVELOPMENT 

Isolated  investigations  of  the  problem  of  quantification  of  human  per¬ 
formance  in  developing  weapon  systems  surfaced  as  a  result  of  the  review  of 
the  literature.  Certain  of  these  may  have  application  to  the  present  effort 
and  are  identified  for  consideration. 

A  detailed  examination  of  this  topic  was  undertaken  by  a  1964  symposium 
on  quantification  of  human  performance.  The  December  1964  Human  Factors 
.lournal,  a  special  issue,  was  devoted  to  selected  (and  edited)  papers  pre¬ 
sented  at  the  symposium  and  workshop  held  at  Albuquerque,  New  Mexico,  in 
August  1964.  This  issue  contains  a  theoretical  article  by  Leuba  (1964) 
dealing  with  human  parameter  quantification  methods.  In  this  paper,  Leuba 
provided  examples  of  the  types  of  quantifications  required  in  man-machine 
systems.  Leuba  associated  selected  equipment  design  procedures  (e.g.  task 
analyses,  trade-off  studies)  with  the  different  types  of  measurement  scales. 

As  example;  he  stated,  "Trade-offs  imply  the  existence  of  ratio  scales. 

This  fact  accounts  for  the  current  difficulty  engineering  psychology  is  ex¬ 
periencing  in  performing  reasonable  trade-off  operations  with  ...  systems 
engineers".  Leuba  then  progressed  to  a  summary  discussion  of  methods  of 
quantification  which  could  be  applied  to  man-machine  system  problems. 

Unfortunately,  neither  Leuba,  nor  any  of  the  other  symposium  participants 
could  suggest  a  procedural  solution  to  quantification  of  human  performance 
factors  in  system  development.  Ten  years  later,  a  suitable  methodology  for 
providing  answers  is  still  needed.  As  DeGreene  (1974?)  states  "To  date  there 
has  been  no  cohesive  examination  of  the  interrelationship  between  new 
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technology  and  manpower  resources.  ...abilities  and  skills  must  be  better  _ 
related  to  tasks.  Jobs,  and  occupations.  This  is  the  ongoing  challenge  to 
human  factors." 

In  this  last  decade,  a  number  of  approaches  to  the  problem  of  quantify¬ 
ing  human  performance  factors  have  been  attempted.  Rook  (1964)  reported  on  a 
method  for  evaluating  system  failure  performance  based  on  rank-order  data. 

Irwin,  Levltz  and  Freed  (1964)  reported  on  the  application  of  a  sub¬ 
jective  method  of  obtaining  human  reliability  data.  The  approach  Involved 
the  use  of  a  Likert-type  scale  to  obtain  reliability  estimates  for  60 
maintenance  tasks.  In  this  approach,  the  authors  related  their  findings  to 
performance  measures  for  operation  of  electronic  equipment  wi.lch  were  developed 
by  American  Institutes  for  Research  (Payne  et  al.  1962). 

Kaplan  (1967)  reported  on  the  results  of  investigating  several  types  of 
scales  for  obtaining  value  Judgments  from  subject  matter  experts.  Ratio, 
difference,  direct,  and  paired  comparison  techniques  were  selected  for  study. 
The  author  selected  the  difference  scale  based  on  the  fact  that  it  had  a 
standard  which  served  to  stabilize  the  scale  in  the  absence  of  a  known  di¬ 
mension  for  the  measure  being  considered. 

Finley  et  al  (1970)  performed  a  technical  review  of  human  perfornuince 
prediction  in  man-machine  systems.  As  a  part  of  this  effort,  over  500 
performance  prediction  tests  were  reviewed.  The  authors  felt  that  develop¬ 
ment  of  a  usable  human  performance  prediction  technique  dictat.-d  adherence  to 
certain  essential  theoretical  .ind  methixlol leal  requirements  which  they 
identified  as: 

1.  Accomplishment  of  a  more  exact  understanding  and  description  of 
system  and  behavior ial  phenomena. 

2.  Recognition  tliat  the  prediction  problems  are  multi-dimensional  and 
multi-level . 

3.  Utilization  of  modern  test  development  techniques  (e.g.  ut 1 1  it v 
analysis. ) 

The  final  conclusion  reached  by  the  autliors  echoed  the  general  concensus 
found  in  earlier  research,  "Human  performance  prediction  tests  can  serve  a 
great  potential  future  role  in  the  understanding,  prediction,  and  control  of 

human  performance  in  man-machine  systems;  but  only  to  the  degree  that  manv 
current  theoretical  and  methodological  problems  are  resolved. 
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Sect  ion  1 1 1 


SUMMARY 

This  riview  of  literature  addressed  the  topics  of  technological 
forecasting  methods  und  techniques  for  assessing  the  effect  of  technological 
advances  on  human  resources.  The  guiding  philosophy  was  to  Identify  techniques 
and  measures  which  could  be  applied  in  seeking  resolution  to  the  problem  of 
specification,  in  advance  of  weapon  system  development,  of  the  effects  of 
technological  innovations  on  the  human  resources  required  to  maintain  and 
operate  the  weapon  system,  and  of  producing  this  data  in  forms  useful  to  system 
designi-rs  and  to  personnel  planners. 


In  order  to  provide  this  substantive  analysis,  a  set  of  evaluative 
criteria  was  generated  against  which  the  Identified  studies  were  judged. 
These  criteria  were: 


1.  Does  the  method  have  application  to  the  present  research  problem? 

2.  Is  the  method  applicable  in  total,  or  are  only  selected  elements 
appropriate? 

3.  Does  the  method  produce  results  couched  in  quantitative  terms? 

•4.  What  limitations  are  imposed  by  the  method  when  considering  such 

variables  as 

—  cost 

—  t  Ime 

—  ease  of  application 

—  computer  feasibility? 

5.  Are  there  any  evaluative  criteria  associated  with  the  proposed 
method;  for  example,  can  statements  be  made  concerning 

—  probability  of  success  with  method 
levels  (and  limits)  of  confidence 

—  validity  of  the  procedure? 


The  review  of  the  lit 
applied  hi  t  to  produce 
However,  the  review  of  the 
"Tr(  lino  I t)p,  i c.i  I  l•'o|•t‘,■,lsl  inp. 


erature  failed  to  reveal  a  procedure  which  could 
the  answers  sought  under  this  research  project, 
methods  and  techniques  subsumed  under  the  topic 
did  suggest  possible  approaches  having  merit  I 


be 


or 


the  prcsisK  task. 


1 echno loglca  I  torecastlng  is  a  young  and  vibrantly  active  field 
attracting  participants  from  many  professional  disciplines.  The  sheer 
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number  of  practitioners  and  their  resultant  documentation  compound  the  task 
of  objective  evaluation  of  the  merits  of  the  proffered  approaches.  However, 
among  those  attempting  to  appraise  technological  forecasting,  a  consensus 
was  apparent  favoring  a  normative  approach  to  forecasting.  This  approach 
was  felt  to  permit  the  reduction  of  an  objective  to  a  series  of  smaller 
choice  considerations  which  were  more  responsive  to  some  form  of  quantifi¬ 
cation. 

Review  of  the  available  normative  techniques  to  technological  forecast¬ 
ing  led  to  the  conclusion  that  a  technique  amenable  for  study  of  the  problem 
under  Investigation  is  a  modification  of  the  relevance  tree  method  known  as 
a  Design  Option  Decision  Tree.  The  relevance  tree  approach  has  been  demon¬ 
strated  to  be  a  useful  vehicle  for  carrying  out  successive  identification  of 
Increasingly  finer  components.  A  basic  strength  of  this  method  lies  in  the 
fact  that  it  forces  an  organization  and  structuring  of  the  problem  and  helps 
to  assure  a  complete  consideration  of  alternative  methods  for  solution  of 
the  topic  under  study.  With  the  design  option  decision  tree,  any  number  of 
solutions  to  the  overall  problem  can  be  traced  and  evaluated  in  terms  of 
their  requirements  on  the  resources  and  technologies  required  for  implemen¬ 
tation.  Lastly,  the  use  of  this  method  provides  an  easy  means  of  determin¬ 
ing  If  a  specific  solution  has  been  overlooked. 

The  design  option  decision  tree  procedure  must  have  some  method  associ¬ 
ated  with  it  for  quantitatively  assessing  the  Impact  of  the  technologies  re¬ 
flected  at  various  points  in  the  tree  on  the  human  resource  parameters  which 
would  be  involved  with  each  decision  option. 

Since  no  solution  to  the  requirement,  quantizing  the  impact  of  techno¬ 
logical  innovations  on  human  resource  parameters,  was  revealed  as  a  result 
of  the  literature  search,  it  is  felt  that  while  the  DODT  will  provide  con¬ 
siderable  insight  into  the  system  problem,  some  other  methodology  must  be 
used  in  association  with  it  to  provide  the  data  required.  The  few  studies 
located  ancillary  to  this  topic  investigated  selected  psychometric  tech¬ 
niques  but  offered  no  procedural  solutions.  The  selection,  or  development, 
of  a  method  to  quantize  human  resource  impact  predictions  must  be  an  empha¬ 
sized  goal  of  the  present  study. 

Attempts  to  provide  assurance  of  the  utility  and  validity  of  any  tech¬ 
nique  developed  in  this  study  must  be  included  as  a  part  of  the  overall  study . 
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As  suggested  by  Roberts  (1969),  and  in  part  employed  in  an  earlier  study 
(Purvis,  Mallory  and  McLaughlin,  1965),  an  inherently  attractive  approach  to 
estimate  the  validity  of  any  proposed  technique  would  appear  to  be  the  com¬ 
bination  of  an  experential  model  and  a  forecasting  model  into  an  overall 
procedure  termed  a  learning  model  by  Roberts  but  perhaps  more  accurately  re¬ 
ferred  to  as  an  "Adaptive  Model."  In  brief,  this  model  would  combine  a  real¬ 
time  system,  on  which  human  resources  data  would  be  collected  and  Interpreted, 
with  a  forecasting  model.  Human  resources  data  for  the  real-time  system  also 
would  be  generated  by  the  forecasting  model.  Comparison  of  the  two  data  sets 
would  permit  modification  of  the  forecasting  model  until  a  parallelism  of  data 
was  achieved.  At  this  point  the  forecasting  model  could  be  employed  with 
specific  areas  of  technological  advance. 

Another  approach  synthesized  from  the  lit.^rature  review  might  be 
classified  as  an  "Incremental  Approximation"  method.  This  approach  is  sug¬ 
gested  by  the  procedure  followed  in  studies  reported  by  Crossman  and  cohorts 
(1966  and  1969)  (see  Appendix  A-12  and  A-17).  The  procedure  basically  would 
represent  a  refinement  to  the  adaptive  model  proposed  in  the  last  paragraph. 

This  approach  is  conceptualized  as  entailing  the  selection  of  a  tech¬ 
nology  area  which  has  experienced  recent  application  of  some  technological 
Innovation.  The  area  selected  must  have  available  instances  of  both  the 
static  and  the  innovative  technology  in  operation.  Empirical  data  is  collected 
on  the  operation  employing  static  technology.  The  normative  predictive  pro¬ 
cedure  and  quantification  method  is  independently  applied  to  this  same  tech¬ 
nology  and  refined  as  described  under  the  adaptive  model.  The  revised 
prediction  procedure  is  then  applied  against  the  operation  Incorporating  the 
innovative  technology  and  the  results  validated  against  empirical  data  col¬ 
lected  from  field  operational  experience  with  the  innovative  technology. 

After  making  any  indicated  refinements,  the  prediction  procedure  is  then 
ready  for  application  to  conceptual  systems  employing  anticipated  technology. 

Reiterated,  the  overall  research  goal  of  this  study  is  the  development 
of  a  procedure  whereby  a  family  of  existing  or  newly  generated  techniques 
and  measures  can  be  applied  to  determine  in  advance  of  actual  system  develop¬ 
ment,  the  impact  of  new  technology  on  the  human  resources  required  for 
operation  and  maintenance  of  the  system. 
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The  next  task  in  the  conceptualized  study  approach  is  envisioned  as  the  most 
difficult  undertaking  of  the  entire  effort.  This  view  finds  support  from  a 
number  of  sources  (Ltiuba,  1964;  Crossman,  1965b;  Finley,  et  al,  1970;  and 
DeGreene,  19747).  Specifically,  this  next  task  is  that  of  developing 
measures  and  techniques  permitting  satisfactory  levels  of  quantification  of 
the  impact  of  specific  technological  Innovations  on  selected  human  resource 
parameters.  The  avenues  of  approach  requiring  pursuit  are: 

•  modification  of  an  existing  technique  for  human  performance  prediction 
to  make  it  amenable  to  the  present  requirement . 

*  integration  of  two  or  more  existing  techniques  to  result  in  a  syn¬ 
thesis  having  applicability  to  the  prediction  task. 

'  development  of  totally  new  techniques  or  measures. 

The  level  of  success  achieved  in  this  task  will  influence  the  remain¬ 
ing  study  objectives:  The  application  of  the  resulting  measures  using  the 
approaches  suggested  earlier  in  this  report,  and  an  evaluation  of  the  value 
and  utility  of  the  procedure  to  system  designers  and  planners. 

The  absence  of  objective  criteria  for  assessing  the  procedure  dictates 
a  reliance  on  expert  judgment.  The  use  of  such  a  subjective  source  of  eval¬ 
uation  requires  great  care  in  the  shaping  and  conduct  of  the  information 
gathering  process.  This  fact  is  clearly  recognized  and  as  the  study  pro¬ 
gresses,  attention  will  be  directed  to  any  plausible  way  of  reducing  the 
subjectivity  of  the  process. 

The  authors  are  intimately  aware  of  the  exploratory  and  feasibility  na¬ 
ture  of  the  present  undertaking.  However,  the  importance  of  this  approach 
should  not  be  underestimated.  As  indicated  by  Darracott  et  al  (1967),  "... 
the  current  technology  race  places  a  high  premium  on  the  ability  to  assess 
developing  technological  trends  correctly.  The  pace  of  this  race  and  the 
increasing  complexity  of  the  problem  have  exceeded  the  ability  of  military 
forecasters  to  assess  these  trends  by  Intuitive  methods  alone."  In  the 
remaining  stages  of  this  study,  selected  avenues  of  approach  will  he  pursued 
in  an  attempt  to  develop  an  acceptable  method  permitting  a  system  designer 
to  conceptualize  the  probable  impact  of  and  manipulate  values  for  selected 
human  resource  components  in  designing  cost-effective  advanced  military 
systems.  Should  no  method  proposed  in  this  study  result  in  a  procedural 
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soluLion  to  this  decades  old  problem,  it  is  believed  that  a  contribution 
nevertheless  will  have  been  made  by  this  endeavor  in  that  other  researchers 
will  be  able  to  stand  on  our  shoulders  in  their  continuing  search  for  a 
solution. 
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rho  importance  of  human  factors  to  system  performance  has  been  shown  a 
number  of  times.  Both  logically  and  empirically,  the  negative  effect  of  In¬ 
adequate  consideration  of  the  human  element  during  design  can  be  demonstrated. 
The  contribution  of  human  error  to  the  unreliability  of  overall  system  per¬ 
formance  has  been  graphically  and  empirically  illustrated.  Evidence  is 
accumulating  that  the  cost  of  the  personnel  to  operate  and  maintain  a  system 
throughout  its  useful  life  Is  equal  to  or  exceeds  that  of  the  hardware.  Thus, 
we  have  a  double-edged  problem,  performance  decrement  and  high  costs,  which 
can  be  related  to  the  human  element  of  the  system.  This  problem  has  been 
made  more  severe  by  a  history  of  development  of  new  systems  with  emphasis 
primarily  on  the  design  of  iiardware  and  with  little  or  no  regard  for  the 
lap.ibility  or  cost  of  tlio  personnel  that  will  be  available  to  support  the 
system.  It  has  been  advocated  that  one  way  to  reduce  the  problem  is  to  de- 
velop  systems  with  liumaii  lactors  data  included  as  design  re(|u irements .  This 
study  is,  therefore,  directed  toward  a  controlled  examination  of  the  utility 
ol  I KI)  (Personnel  Resources  Data)  in  system  design  and  to  the  determination 
of  the  conditions  under  which  PRD  can  be  effective  In  influencing  that  design. 

GENERAL  RESEARCH  APPROACH 

The  major  purpose  of  this  study  was  to  determine  the  effect  on  system 
design  of  using  manpower  and  personnel  resources  data  as  design  requirements. 
Secondary  objectives  were  to  determine  under  what  conditions  and  in  what 
form  these  data  should  be  used  to  have  maximum  effect  on  design.  Equipment, 
manpower  data  (e.g.,  quantities  and  skill  levels),  and  PRD  Inputs  (e.g.,  task 
information)  which  were  produced  during  the  development  of  the  Titan  III 
propellant  transfer  and  pressurization  subsystem  were  adopted  and  presented 
Incrementally  to  six  design  engineers  to  simulate  the  Air  Force  phase  lA/lB 
development  of  that  subsystem.  Subjects  were  required  to  create  schematics, 
equipment  ilescrlpt ions  and  drawings,  control  panel  layouts,  operating  pro¬ 
cedures  .ind  l)iils  of  material.  Gost-ef  feet  iveness  measures  including 
i'c|uipmi'ut  ciist,  equipmiuit  reliability,  human  rt'l  iabi  1  ity,  system  safety  and 
ilesi);n  a(le(|uai  v  were  api)|ied  to  the  data. 

ANALYSIS 


The  research  strategy  developed  by  Meister  and  Farr  (1966)  and  Meister 
and  Sullivan  (1967)  involves  placing  the  engineer  in  a  realistic  design  situa¬ 
tion  in  which  he  must  solve  a  series  of  design  problems  by  using  Informational 
inputs  related  to  these  problems.  In  adapting  this  general  methodology  to 
the  present  study,  the  following  steps  were  performed: 

1.  Selection  of  an  already  operational  subsystem  which  could  serve  as 
a  model  subsystem  for  the  development  of  test  inputs. 

2.  Selection  of  appropriate  engineer-subjects  skilled  in  design  of  tlie 
type  of  subsystem  selected. 
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3.  Determination  of  the  equipment  and  PRD  Inputs  which  are  character¬ 
istically  provided  during  the  system  definition  phase  of  development. 

4.  Development  of  manpower  and  person  lel  resources  data  Inputs. 

5.  Determination  of  the  sequence  In  which  these  Inputs  should  be 
provided. 


6.  Determination  of  the  design  responses  and  outputs  which  the  engineer- 
subjects  should  supply  in  attempting  to  solve  the  design  problems. 

7.  Determination  of  specific  measures  which  could  be  used  to  answer 
the  questions  which  Initiated  the  study. 

FINDINGS 


1.  Manpower  requirements  and  personnel  resources  data  inputs  do  have  an 
Influence  on  the  equipment  configuration,  but  this  Influence  is  attenuated  by 
a  complex  of  factors  such  as  the  engineer's  indifference  to  and  inability  to 
Interpret  human  factors  considerations  meaningfully;  and  more  importantly,  by 
the  Inadequate  timing  of  PRD  Inputs. 

2.  The  potential  Influence  of  personnel  requirements  and  PRD  Inputs  on 
the  equipment  configuration  Is  much  greater  than  Is  presently  achieved.  PRD 
Inputs  would  exercise  much  greater  effect  If  they  were: 

a.  phrased  as  specific  design  requirements  and  constraints  and  in¬ 
cluded  in  the  SOW  (Statement  of  Work) . 

b.  phrased  In  concrete  design-relevant  terms. 

3.  The  manner  in  which  the  engineer  designs  has  a  significant  effect 
upon  his  reaction  to  personnel  requirements  and  his  use  of  PRD  inputs,  hence 
on  their  Influence  on  the  subsystem  configuration.  The  engineer's  design 
concept  is  so  quickly  developed  that  traditional  timing  of  manpower  require¬ 
ments  and  PRD  Inputs  lag  that  concept.  The  engineer  resists  any  change  to 
his  initial  concept  as  a  restriction  on  his  freedom  to  design  creatively. 

4.  The  Importance  of  supplying  meaningful  manpower  information  as  design 
requirements  in  the  SOW  and  subsequent  QQPRI  analyses  by  trained  human  factors 
personnel.  Is  highlighted  by  the  fact  that  the  engineer's  manning  concept  for 
his  own  design  does  not  appear  to  correspond  to  the  needs  of  his  subsystem 
designs. 


5.  The  results  of  this  study  are  in  accordance  with  those  of  previous 
design  engineering  studies.  This  increases  the  confidence  one  can  feel  in 
the  conclusions  derived. 

CONCLUSION 


It  was  found  that  manpower  requirements  and  PRD  inputs  do  influence  the 
equipment  configuration,  but  in  this  study  only  moderately,  bec.ause  the 
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equipment  design  proceeded  so  rapidly  that  Incremental  PRD  Inputs  Inevitably 
lagged  the  design.  Engineers  were  responsive  only  to  Inputs  which  were 
framed  as  design  requirements  and  which  were  Interpreted  In  design-relevant 
terms.  Confirming  the  results  of  previous  studies,  engineers  were  found  to 
be  generally  unaware  of  or  Indifferent  to  personnel  considerations.  Differ¬ 
ent  engineers  Interpreted  the  same  design  requirements  and  assigned  priority 
to  design  criteria  differently.  The  engineers  relied  heavily  on  experience 
and  stereotyped  solutions  for  design  answers.  The  results  of  the  study 
indicate  that  manpower  and  personnel  analyses  must  be  performed  prior  to  the 
Issuance  of  a  Request  for  Proposal  (RFP)  and  not  delegated  to  the  develop¬ 
ment  contractor.  The  contractor  must  be  required  to  design  to  a  detailed 
manning  structure  which  Is  specified  In  his  Statement  of  Work.  Further 
reconinendatlons  are  supplied  which  suggest  ways  in  which  Air  Force  management 
of  the  personnel  subsystem  program  should  be  revised. 


Martino,  Jo«»eph  P. 


Technological  Forecasting  for  Decisionmaking 


A- 2 


This  book  is  intended  to  provide  an  advanced  treatment  of  technological 
forecasting.  It  is  a  "how  to"  book:  how  to  do  technological  forecasting, 
how  to  apply  it  in  specific  decision  situations,  and  how  to  avoid  some  of  the 
more  common  errors  and  difficulties  in  the  preparation  of  individual  forecasts 
Although  this  abstract  lists  the  chapter  titles,  it  is  primarily  concerned 
with  chapter  9,  "Normative  Methods." 

GENERAL  RESEARCH  APPROACH 


The  contents  by  chapter  number  are  as  follows: 


1. 

Introduction 

11. 

Research  and  Develop¬ 

IH. 

Technological  Fore 

ment  Planning 

casting  in  .Social 

2. 

Delphi 

12. 

Research 

Decisions 

3. 

Forecasting  by  Analogy 

13. 

Technology  Advance¬ 

19. 

Some  Common  Fore¬ 
casting  Mistakes 

4. 

Growth  Curves 

ment 

20. 

Dangers  of  Nega¬ 

5. 

Trend  Extrapola¬ 

14. 

Product  Development 

tive  Forecasting 

tion 

15. 

Test  and  Evaluation 

21. 

Evaluating  Fore¬ 

6. 

Analytical  Methods 

16. 

Technological  Fore¬ 

casts  as  Decision 
Information 

7. 

Breakthroughs 

casting  in  Business 
Decisions 

22. 

Presenting  the 

8. 

Combining  Forecasts 

17. 

Technological  Fore¬ 

Forecast 

*9. 

Normative  Methods 

casting  in  Government 

23. 

Tlie  Future  of 

Decisions 

Technolog ica 1 

10. 

Planning  and  Decision¬ 
making 

Forecasting 

*Chapter  9  covered  in  this  abstract. 

The  objective  of  the  normative  methods  is  to  determine  the  technological 
capability  which  will  be  required  to  carry  out  some  function,  based  on  an 
estimated  or  projected  demand.  Exploratory  and  normative  methods  are  not 
competitive  with,  or  replacements  for,  one  another.  Both  are  essential,  and 
both  must  be  used  together.  Normally  one  does  not  bother  to  prepare  an 
ex  iratory  forecast  of  some  technology  unless  there  is  a  normative  forecast 
(ac  least  an  implicit  one)  that  the  technology  will  be  needed.  Likewise,  one 
does  not  normally  prepare  a  normative  forecast  without  some  idea  that  it  will 
be  possible  to  meet  the  goals. 

ANALYSIS 


The  three  most  common  methods  of  normative  forecasting  are  relevance 
trees ,  morphological  models,  and  mission  flow  diagrams.  Relevance  trees  are 
used  where  the  system  or  process  being  analyzed  can  be  described  in  terms  of 
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levels  of  causation,  levels  of  complexity,  or  levels  of  hierarchy.  Morpho¬ 
logical  models  are  used  where  the  system  or  process  can  be  broken  down  into 
elements  which  can  be  altered  independently,  and  where  there  are  no  hierarchi¬ 
cal  relationships.  Mission  flow  diagrams  are  used  where  the  system  or  process 
can  he  described  in  terms  of  one  or  more  paths  of  sequential  steps. 

1 .  Relevance  Trees 


Relevance  trees  are  used  to  analyze  systems  or  processes  in  which  distinct 
levels  of  complexity  of  hierarchy  can  be  Identified.  They  are  developed  by 
carrying  out  successive  Identification  of  Increasingly  finer  components,  at 
progressively  lower  levels.  Each  branch  divides  (at  a  node)  into  two  or  more 
branches  at  the  next  lower  level.  Each  node  is  a  goal  for  all  those  branches 
depending  from  it.  Each  goal  is  satisfied  by  the  achievement  of  all  branches 
below  it  and,  in  turn,  derives  its  validity  as  a  subgoal  from  the  sequence  of 
branches  linking  it  with  the  top  of  the  tree.  Relevance  numbers  can  be  as¬ 
signed  to  each  of  the  brandies  of  the  tree  and  an  overall  figure  of  merit  can 
he  associated  witli  a  total  patii  t  hrougli  the  tree. 

“ •  Morphological  Models 

"Morphology"  describes  the  s^udy  of  the  form  or  structure  of  the  entity 
of  interest  (l.e.,  plants  and  animals  in  biology,  rocks  in  geology).  The 
essence  of  the  morphological  method  of  analysis  is  to  break  a  problem  down 
into  parts  which  can  to  some  extent  be  treated  Independently,  with  several 
solutions  or  approaches  to  each.  A  system  or  problem  is  broken  down  into 
parts  which  can  be  treated  independently,  and  as  many  solutions  or  approaches 
as  possible  are  devised  for  each  part.  The  technological  performance  re¬ 
quired  for  each  solution  is  then  obtained,  and  this  is  used  as  a  goal  for  the 
technology  involved.  Some  of  the  solutions  can  be  rejected  immediatelv  on 
the  grounds  of  feasibility;  others  can  be  rejected  because  of  conflicts  with 
other  solutions.  One  is  finally  left  with  a  set  of  feasible  goals  for  the 
technologies  Involved. 

)•  Diagrams 

I'he  mission  I  low  diagram  was  originally  devised  as  a  means  of  analyzing 
lert.iin  tvpes  ol  military  missions.  However,  the  technique  can  be  apj)licd 
to  the  analysis  of  anv  sequential  process.  The  technique  involves  mapping 
all  the  alternative  routes  or  sequences  by  which  some  mission  or  task  can 
be  accomplished.  All  significant  steps  are  identified  on  each  route.  It 
then  becomes  possible  to  identify  the  difficulties  and  costs  associated  with 
each  route.  In  addition,  it  is  possible  to  create  new  alternative  routes, 
and  identify  the  difficulties  and  costs  associated  with  these.  Once  these 
difficulties  and  costs  are  identified,  performance  requirements  can  be  de¬ 
rived  for  the  technologies  Involved,  and  used  as  normative  forecasts. 

FINDINGS 

1.  Normative,  or  goal-setting  methods  of  fort'casting  are  applied  to 
determine  the  level  of  functional  capability  which  must  be  achieved  if  some 
problem  is  to  be  solved  or  some  difficulty  overcome. 
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2.  Relevance  trees  are  used  to  carry  out  analysis  of  hierarchical 
structures. 

3.  Morphological  models  are  used  to  analyze  parallel  structures. 

Mission  flow  diagrams  are  used  to  analyze  processes  with  steps  In 
a  spatial,  temporal,  or  logical  sequence. 

5.  Often  the  same  system  can  be  analyzed  by  either  a  relevance  tree  or 
a  morphological  model.  In  such  cases,  the  forecaster  should  use  whichever 
method  Is  most  convenient  or  appropriate  for  the  problem  at  hand. 

CONCLUSION 


The  strength  of  normative  methods  is  that  they  tend  to  organise  and 
structure  the  problem.  They  can  help  assure  completeness,  so  that  solutions 
holding  some  promise  are  not  overlooked.  By  systematically  laying  out  t  lie 
structure  of  a  problem,  they  can  assist  In  the  generation  of  new  alternatives 
which  may  be  superior  to  those  currently  in  use.  Even  If  all  the  alternatives 
uncovered  with  the  use  of  normative  methods  prove  inferior  to  those  already 
known,  this  alone  gives  additional  confidence  in  the  choice  of  one  of  the 
methods  as  superior  to  the  rest. 

It  Is  essential  that  the  normative  methods  be  kept  in  proper  perspective. 
They  are  In  no  sense  a  substitute  for  creativity  or  imagination.  They  merely 
provide  a  systematic  means  of  examining  technological  requirements. 
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Hacke,  James  E.  Jr., 


The  Feasibility  of  Anticipating  Economy p  and  Social 
Consequences  of  a  Major  Technological  llinovat ion 

Forecasting  is  methodologically  distinct  from  goal  formulation  or 
fanning  and  Is  as  methodologically  Justified  In  its  use  of  the  results  of 

forforeLstinr^U  »"  analyst,,  ths  methods  used 

-,nH  ‘'losses,  those  for  forecasting  patterns 

avallaMe  ^^^SSSiars.  with  five  generlc.lly  dlttere^T^d, 

GENERAL  RESEARCH  APPROArn 

InvesM^tlornreh  description  end  examples,  an 

-tigation  of  the  assumptions  underlying  its  use,  an  estimate  of  its 

are'Ln  "  ‘delineation  of  its  ll^itatioL  ThLe  Lthods 

forecLsUng  technological,  demographic,  economic,  and  social 

Existing  literature  on  past  economic  and  social  consequences  of  maior 

tion-°ih®  f  -1  narration  and  sJatistical  deLrip- 

changef^suianrfror  following  an  innovation  and 

Changes  resulting  from  an  Innovation.  To  make  this  distinction,  one  must 

been"made  TheTu  have  been  the  course  of  events  if  the  innovation  had  not 
een  made.  The  methods  of  forecasting  can  be  applied  equally  well  to  this 
delineation  of  hypothetical  consequences. 

ANA_LYS;^I_S 

tion  s^Ltrilt^  social  consequences  of  an  Innova- 

tlon  starts  with  forecasts  of  economic  and  social  conditions  over  the  period 

ot  me  involved  and  forecasts  in  their  light  the  probable  level  of  effort 
nrobiM  0  expended  in  developing  the  innovation.  Then  one  forecasts  the 
probable  pattern  of  its  development  and  formulates  and  forecasts  the  changes 

nribr^'^r  of  its  performance.  Having  this  descrijffon  rfhe 

inbable  future  tech  lological  development  of  the  innovation,  one  can  forecast 
probable  economic  and  social  Impacts.  Since  these  impacts  modify  the  forecast 
economic  and  social  conditions  with  which  one  starts,  anticipating  economic 
and  social  consequences  of  an  Innovation  is  in  principle  an  iterative  process. 

that  assumption 

that  methods  of  assessing  economic  and  social  Impacts  of  innovations  were 

uch  further  developed  than  were  those  of  technological  forecasting.  How¬ 
ever,  the  reverse  proved  to  be  the  case. 

2.  A  survey  and  analysis  of  the  literature  on  technological  (and  other) 

).  A  fundamental  approach  was  to  describe  relevant  portions  of  the 
economy  and  the  society  following  introduction  of  the  InnLation  and  [o 
escribe  what  these  might  have  been  had  the  innovation  not  been  introduced. 
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llfterence  between  these  two  descriptions;  the 
difference  was  a  measure  of  impact  of  the  innovation. 

findings 

is  as  justifiable  epistemologically  as  engineering 
cine,  or  any  other  discipline  using  the  findings  of  empirical  science. 

fn  “ethods  of  forecasting  are  basically  the  same  whether  applied 

to  technology,  demography,  economics,  or  sociology. 

forpcalr/*'®  division  of  forecasting  into  parameter  forecasting  and  pattern 

Ihnrnforn  probuhly 

rh;,na!’  generically  different  methods  of  forecasting 

chanfn  i  patterns  and  five  generically  different  methods  of  forecasting 
changes  in  parameters.  <-1 mg 

5.  Methods  of  forecasting  changes  in  parameters  include  statisticil 
measures  that  give  so.e  Indication  of  the  precision  to  be  expec  ed  i^ei  a 

thIT™™i''“''r  r'  limits  set  by  these  nersire;  .  l^^over 

he  dynamics  of  the  situation  are  changing  and  the  forecast  should  be  redraw,!. 

orsan?!atII;^  °5  '■'“Ses  In  patterns  or  structures  of 

alternftl!  ^  ^  Precision  or  confidence  p-asures  and  do  not  preclude 

tMrUmU^Jon'rrrf  development.  This  conctltutes  the  single  most  Impor- 
tant  limitation  to  the  current  state-of-the-art  in  forecasting. 

social assessment  of  the  past  economic  and 
tSat  ft^^U  .„to°eh  f'”'  "'8  °t  this  study  In 

pares  what  act^ll.  Sld^  '’‘’“‘’t*’'  "p"'  ‘t  com- 

tlornrco:r“Jo^Jstence!"  '"p  ‘""p- 

8.  The  largest  single  practical  difficulty  is  likely  to  be  loriMira 
adequate  data  for  analysis.  Data  on  the  transistor  through  1956  were  not 
extensive  enough  to  allow  projection  to  1966.  In  the  absence  of  data  over  a 

Jud|nent“*  trends,  one  must  rely  solely  on  Informed 

a  feasibility  of  anticipating  economic  and  social  consequences  of 

development  of^thPri  depends  strongly  on  state  of  technological 

oevetopraent  of  that  innovation.  Data  are  scarce  in  the  early  stages  of 

«“:u1j^J"fs-nd‘r“r:!rr  whst'L1^^::ntIon 

natuJ^of^I’LT'‘“'/°'’'“"“"*  'PPhnlques  used  ere  determined  by  the 

£nl“lT 

rapid  transit  would  require  other  kind,  of  data.  Availability  of  the 
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required  data  would  determine  feasibility  of  producing  a  meaningful  anticipa¬ 
tion  of  future  economic  and  social  Impacts. 

11.  Although  the  forecast"  of  the  Impact  of  the  transistor  and  the 
forecasts  suggested  are  useful  as  demonstrations,  such  forecasts  would  be 
of  more  value  to  planners  if  they  were  continuous  processes.  It  would  be 
particularly  valuable  to  racaitor  a  course  of  development  that  has  been 
forecast  to  pinpoint  when  that  course  diverges  from  expectations  and  to 
obtain  a  more  rapid  check  than  is  now  possible  on  forecasting  methods. 

amaiisjoN 

It  is  feasible  to  anticipate  the  economic  and  social  consequences  of  a 
major  technological  innovation  with  sufficient  detail  and  reliability  to  be 
of  major  assistance  to  the  long-range  planner. 

It  is  not  feasible,  iiowever,  to  forecast  quantitatively  the  trends  in 
a  parameter  or  a  pattern,  with  the  certainty  that  the  outcome  will  fall 
within  stated  confidence  limits.  Such  certainty  would  require  both  exact 
knowledge  ot  the  mathematical  form  of  the  trend  curve  and  complete  assurance 
that  no  perturbation  of  the  trend  will  occur  in  the  interim. 


Anticipating  trends  in  the  characteristics  of  a  major  technological 
Innovation  is  feasible.  It  is  feasible  to  anticipate  some  of  them  with 
enough  precision  and  reliability  for  the  forecast  to  be  of  significant  value 
to  the  planner.  The  precision  and  reliability  achievable  as  a  function  of 
time  may  match  what  the  planner  needs  to  know  at  that  time. 

On  the  basis  of  these  anticipated  trends  in  the  technical  character¬ 
istics  of  an  Innovation  and  of  its  supply  and  cost,  it  is  possible  to  antici¬ 
pate  some  of  its  economic  and  social  consequences.  In  essence,  this  con¬ 
sists  in  comparing  the  probable  future  course  of  development  in  the  economy 
and  the  society  with  what  would  be  likely  if  the  innovation  had  never 
appeared.  Demographic,  economic,  and  social  forecasting  are  feasible  to 
the  same  extent  that  technological  forecasting  is. 
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Roberts,  Edward  B.  Exploratory  and  Normative  Technological  Forecasting; 

A  Critical  Appraisal  ~ 

(This  report  was  contained  as  an  appendix  in  the  Marvin  J.  Cetron  book, 
Technological  Forecasting  -  A  Practical  Approach.) 

Increased  recognition  during  the  present  decade  of  the  importance  of 
science  and  technology  to  corporate  and  national  existence  has  produced  an 
Intensive  search  for  new  methods  for  managing  research  and  development.  The 
attention  being  devoted  to  so-called  "technological  forecasting"  is  one 
manifestation  of  this  concern, 

GENERAL  RESEARCH  APPROACH 

This  report  presents  a  critical  appraisal  of  the  field  of  technological 
forecasting.  The  central  theme  is  that  the  two  phases  of  exploratory  and 
normative  approaches  are  out  of  step  with  each  other.  Exploratory  techniques 
are  too  naive  and  do  not  take  advantage  of  what  has  been  learned  about  fore¬ 
casting  in  nontechnical  areas.  Nor  do  the  exploratory  techniques  reflect 
what  is  known  about  the  influences  upon  the  generation  of  future  technology. 
Normative  techniques  in  contrast  are  too  complex  and  mathematically  sophisti¬ 
cated  and  cannot  justify  their  elegance  on  substantive  grounds. 

ANALYSIS 

Technological  forecasting,"  as  defined  by  those  claiming  to  be  its 
practitioners,  is  actually  two  fields,  joined  more  by  a  vision  than  a  reality. 
On  the  one  hand  is  "exploratory  forecasting,"  the  attempt  to  predict  the 
technological  state-of-the-art  that  will  or  might  be  in  the  future,  or  as 
Cetron  puts  it,  ...a  prediction  with  a  level  of  confidence  of  a  technical 
achievement  in  a  given  time  frame  with  a  specified  level  of  support."  Most 
laymen  assume  that  all  of  "forecasting"  is  this  kind  of  forecasting.  The 
second  aspect  has  been  called  "normative  forecasting"  and  includes  the  organ¬ 
ized  attempts  to  allocate  on  a  rational  basis  the  money,  manpower,  and  other 
resources  that  might  affect  the  creation  of  tomorrow's  technological  state- 
of-the-art.  Normative  forecasting  presumably  provides  aids  to  budgetary  de¬ 
cisions  in  the  technological  area.  Still  more  broadly  defined  by  some  of  its 
leading  exponents,  "normative  forecasting"  applies  to  a  wide  variety  of 
attempts  to  determine  policies  and  decisions  that  will  influence  the  effective 
growth  of  science  and  technology,  in  the  corporation,  the  government  agency, 
or  the  nation  as  a  whole. 

Exploratory  technological  forecasting  includes  a  variety  of  techniques 
for  predicting  the  future  state  of  science  and  technology.  Unfortunately, 
most  of  the  methods  are  really  only  variants  on  simple  trend  extrapolation 
procedures,  broadly  defined,  that  have  limited  utility  in  today's  rapidly 
shifting  technological  environment.  The  principal  exploratory  methods  are: 

1.  So-called  "genius"  forecasting,  based  either  on  individual  wisdom 
or  on  a  group  'genius"  forecasting  process  known  as  the  Delphi  technique. 
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2.  Formal  trend  extrapolation  to  either  a  straight-line  fit  or  an  S- 
shaped  expectation. 


The  formal  trend  methods  Include  single-curve  projections  as  well  as 
estimations  based  on  the  envelope  encompassed  by  the  projection  of  a  family 
of  related  curves.  They  also  Include  both  the  extrapolation  of  a  single  time 
series  as  well  as  the  projection  of  lead-lag  relationships  between  two  time 
series,  the  latter  known  as  "precursive  event"  forecasting. 


Normative  technological  forecasting  activities  attempt  to  provide  a 
basis  for  allocating  technology-generating  resources  so  as  to  maximize 
attainment  of  organizational  goals.  The  effectiveness  of  any  normative 
method  depends  upon  the  following  factors: 


1.  Meaningfulness  of  its  treatment  of  goals. 

2.  Correctness  of  its  assumed  relationships  between  allocated  resources 
and  generated  technology. 


3.  Adequacy  of  its  balancing  of  the  resources-to-technology  considera¬ 
tions  against  the  worth  of  goal  fulfillment. 

A.  Implementabll ity  of  the  method.  Including  the  ability  to  acquire 
reasonable  inputs  at  reasonable  costs  as  well  as  the  ability  to  persuade  or¬ 
ganizational  decision-makers  to  use  the  generated  outputs. 

S(mu'  of  tile  leading  normative  forecasting  techniques  are: 

1.  PRc'ilLE  (Programmed  Functional  Indices  for  Laboratory  Evaluation) 

2.  QUEST  (Quantitative  Utility  Estimates  for  Science  and  Technology) 

3.  PATTERN  (Planning  Assistance  Through  Technical  Evaluation  of  Relevance 
Numbers) 


A.  TRW's  PROBE  II 


FINDINGS 


1.  Comparison  of  the  still  evolving  approaches  to  "exploratory"  and 
normative  technological  forecasting  yields  marked  contrasts.  In  particular, 

the  simple  schemes  used  by  those  trying  to  predict  the  technology  of  the  future 
look  pallid  when  matched  against  the  sophisticated  techniques  designed  by 
those  who  are  allocating  the  resources  that  will  create  the  future.  Explora¬ 
tory  technological  forecasts  are  largely  based  either  on  aggregates  of  "genius" 
forecasts  (e.g.,  the  Delphi  technique)  or  on  the  use  of  leading  Indicators 
and  other  simple  trend-line  approaches.  The  practitioners  of  economic  fore¬ 
casting,  in  contrast,  long  ago  recognized  the  need  for  multivariate  systems 
analysis  and  cause-effect  models  to  develop  reliable  predictions. 

2.  Normative  forecasting  is  at  the  opposite  extreme  on  the  sophistica¬ 
tion  scale,  fully  utilizing  Bayesian  statistics,  linear  and  dynamic  programming, 
and  other  operations  research  tools.  Here,  despite  the  uniqueness,  uncertainty, 
and  lack  of  uniformity  of  research  and  development  activities,  each  of  the 
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designers  of  normative  techniques  has  proposed  a  single  format  wholly  quanti¬ 
tative  method  for  resource  allocation.  Along  the  dimensions  of  unjustified 
standardization  and  needless  complexity,  for  example,  the  proposed  R&D  alloca 
tion  methods  far  exceed  the  general  cost-effectiveness  approach  used  by  the 
Department  of  Defense  in  its  program  and  system  reviews. 

3.  For  both  exploratory  and  normative  purposes,  dynamic  models  of  broad 
technological  areas  seem  worthy  of  further  pursuit.  In  attempting  to 
develop  "pure  predictions"  the  explicit  recognition  of  causal  mechanisms  of¬ 
fered  by  this  modeling  approach  seems  highly  desirable.  This  feature  also  has 
normative  utility,  provided  that  the  dynamic  models  are  limited  in  their 
application  to  the  level  of  aggregate  technological  resource  allocation  and 
are  not  carried  down  to  the  level  of  detailed  R&D  project  funding. 

CONCLUSION 


There  is  no  doubt  that  both  kinds  of  "forecasting"  are  necessary  con¬ 
tributors  to  the  technical  planning  process,  and  for  the  military,  as  well 
as  for  most  corporations,  this  technical  input  is  a  critical  ingredient  of 
an  overall  business  plan.  Yet  only  in  theory  but  not  in  fact  have  these 
two  components,  the  exploratory  and  the  normative,  been  Integrated  adequately. 
Erich  Jantsch  expresses  the  logic  of  and  the  need  for  this  integration: 
"Exploratory  technological  forecasting  starts  from  today's  assured  basis  of 
knowledge  and  is  oriented  towards  the  future,  while  normative  technological 
forecasting  first  assesses  future  goals,  needs,  desires,  missions,  etc.,  and 
works  backward  to  the  present. .  .,The  full  potential  of  technological  forecast¬ 
ing  is  realized  only  where  exploratory  and  normative  components  are  joined 
in  an  iterative  or,  ultimately,  in  a  feedback  cycle". 
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Maincs,  Donald  B.  Kst Imat ing  Manning  Requirements  for  Advanced  Systems: 

(lael,  Sidtii  y  A  Survey  of  the  Defense  Industry 

'1  liore  is  an  urgent  need  for  better  use  of  our  manpower.  An  Important 
step  is  the  early  estimation  of  manpower  requirements  for  advanced  weapon 
systems.  A  simple  and  accurate  i  ethod  is  needed  for  predicting  tliese  re¬ 
quirements  as  early  as  possible  to  prevent  a  subsequent  surprising  levy  on 
available  jaen  and  skills. 

CEN ^A I ^  RESEARCH  APPROACH 

This  researt h  was  conducted  in  an  effort  to  develop  techniques  of  esti¬ 
mating  manning  requirements  while  the  feasibility  of  a  future  system  is  being 
studied.  One  portion  of  this  research  program  was  aimed  at  identifying  and 
quant  it ying  the  ri'levant  variables  that  generate  manning  requirements.  One 
goal  was  to  prodiici'  a  multipurpose  estimation  technique,  System  designers 
could  use  such  a  technique  to  predict  manning  requirements  during  the  early 
conceptual  pliase  and  trade-off  manning  requirements  with  other  design  con¬ 
straints  and  requ  i  ri’raent  s  . 

ANALYSIS 

Tlie  defense  industry  was  surveyed  to  determine  their  methods  and  prob¬ 
lems  ot  estimating  manning  requirements.  The  survey  had  three  principal  aims: 

1.  Determine  the  different  wavs  the  defense  industry  copes  with  the 
problem  of  estimating  manning  requirements  for  future  systems. 

2.  Assess  the  ef  f  I'ct  iveness  of  these  methods  if  they  were  used  with  in¬ 
formation  and  data  available  onlv  during  the  conceptual  phase  of  the 
proposed  weapon  svstem. 

3.  Decide  wliich  metliods  are  more  appropriate  to  specific  kinds  of 
weapon  systems. 

I'lie  quest  ionna i  ri’  used  in  the  survey  was  a  five-item,  open-ended  question 
tvjH'.  Respondents  were  asked  to  give  their  name,  field  of  specialization,  job 
title,  and  tlu‘  eompanv  lor  whoiii  t  liev  were  working.  The  following  questions 
were  asked: 

1.  Eor  wliich  svstem(s)  or  subsystem(s)  have  vou  part  ii: ipated  in  the 
estimation  ol'  manning  requirements? 

2.  Describi*  the  tei-hnlqne  you  havi-  used  or  would  use  to  estimate  tlu‘ 
operator  aiui /or  t  he  maint  enance  manning,  requirements  for  future 
systems. 

3.  Which  fivi“  factors  do  vou  consider  to  be  Important  determiners  ol 
the  number  of  men  that  will  be  required  to  support  systems? 
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4.  If  you  were  asked  to  recommend  three  publications  pertalninR  to  tlie 
estimation  of  manninR  requirements  (eitlu  r  t)pi‘r,iLor  or  m.i  i lU enance) , 
which  ones  would  you  suggest? 

5.  Additional  comments. 


FINDINGS 


Questionnaires  were  sent  to  117  people  in  key  defense  industries.  75 
questionnaires  were  returned  (some  completed  by  a  team  of  respondents).  Of 
these,  55  were  fully  completed  and  were  answered  in  a  way  that  was  felt  to  meet 
the  spirit  of  the  survey.  The  principal  findings  of  the  survey  are  as  follows: 

1.  Dissatisfaction  prevails;  most  industries  feel  that  a  new,  uniform 
and  standardized  method  is  desired.  At  present  there  is  no  standard 
method  that  is  applicable  in  the  early  stages  of  weapon  system 
development . 

2.  Predictions  continue  to  be  made  so  late  that  they  are  of  little  use 
in  influencing  design.  Often  a  prediction  is  made  in  a  form  tliat 
does  not  lend  itself  readily  to  making  design  changi's. 

3.  There  are  definite  trends  and  preferences  in  current  use.  Most  re¬ 
spondents  preferred  traditional  task-analytic  methods  for  est im.it ing 
manning  requirements  and  were  guided  largely  by  USAF  technical 
reports. 

4«  The  main d if f erences  were  that  the  people  who  had  access  to  acquisit ioiv 
tlme  data  used  historic  comparison  as  an  estimation  technique,  while 
those  in  the  conceptual  phase  of  development  liad  more  of  a  tendency 
to  use  other  approaches,  viz,  modeling. 

5.  Those  respondents  who  had  experience  with  space  systems  and  communi¬ 
cations  or  radar  nets  used  guides  other  than  USAF  publications; 
those  with  other  system  experience  preferred  the  USAF  guides. 

6.  In  general,  the  reliance  upon  USAF  guides  and  materials  was  heavier 

in  acquisition  time.  In  other  than  acquisition  time,  USAF  guides  were 
used  in  conjunction  with  company  reports 

7.  There  was  one  significant  finding  from  the  statistical  analysis: 
respondents  who  worked  on  space  systems  relied  more  or  mathematical 
and  objective  methods  of  estimation  than  did  those  whose  experience 
was  with  (communication  or  radar)  nets, 

CONCLUSION 

Five  different  approaches  to  estimating  manning  were  reported,  the  most 
popular  being  the  use  of  task  analysis  or  some  combination  of  t.ask  analysis 
with  variations  of  the  other  approaches.  Other  methods  were:  expert  esti¬ 
mation,  historical  comparison,  sovereign  factors, and  mathematical  models. 

The  most  frequently  cited  guides  were  USAF  publications,  but  a  wide  variety 
of  other  texts,  handbooks,  and  technical  publications  were  also  reported. 
Respondent  experience  ranged  over  the  major  weapon  systems,  subsystems,  and 
control  networks. 
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A-') 

Barton,  H.  R. ,  Purvis,  R.  E. ,  A  Queuing  Model  for  Determining  System 

Stuart,  J.  E.  ,  Mallory,  W.  K.  Manning  and  Related  Support  Requirements 

Increasing  need  for  earlier  estimates  of  manning,  skills,  and  training 
requirements  led  to  the  development  of  mathematically  sophisticated  techniques 
capable  of  computing  and  assessing  requirements  at  every  phase  of  system 
development.  Current  methods  aie  largely  intuitive,  rely  on  bookkeeping 
procedures,  and  are  seldom  applicable  at  pre-hardware  stages  of  system 
development.  The  method  in  this  report  fills  a  need  for  making  trade-offs 
when  investigating  alternatives  in  system  design. 

( d'lNE^RAI.  RESEARCH  APPROACH 

The  report  begins  with  an  analysis  of  hardware  functions  and  develops 
human  requirements  in  terms  of  operational  needs  and  service  rates.  Manning 
and  skill  requirements  are  integrated  over  such  factors  as  desired  environ¬ 
mental  demands,  maintenance  concepts  and  procedures,  and  training  require¬ 
ments.  Two  mathematical  techniques,  queuing  theory  and  linear  programming, 
are  used  to  compute  manning  requirements  and  training  needs. 

ANALYSIS 


The  first  step  is  the  development  of  methods  which  enable  engineers  to 
compute  and  assess  (at  a  very  early  time  in  the  systems  design)  the  manning 
and  skill  and  training  lequlrements  for  a  given  system  design (s)  with  ac¬ 
companying  operational  and  maintenance  concepts.  This  report  addresses 
three  areas: 


1.  Identify  significant  system  variables  that  affect  the  requirements 
for  manning,  skills,  and  training. 


Ihe  approach  in  this  area  was  based  on  a  mathematical  model  relating 
significant  variables  associated  with  manning  and  skill  requirements: 

a.  task  Identification  which  determines  skill 

b.  operational  performance  requirements 
The  operational  requirements  were: 


(1)  operational  readiness  specified  for  the  system  —  a  condition 
that  must  be  satisfied. 

(2)  capability  of  each  operational  unit  to  operate  a  total  num¬ 
ber  of  hours  or  hours  per  mission  for  a  certain  number  of 
missions  per  calendar  time-unit. 

2.  ()u.iiitirv  the  informal  ion  about  t  hesi'  variables. 


I'oi  exampli-:  Opical  iona  1  readini*ss  (R)  =  .88 

total  operational  hours  (t^)  =  16 
number  of  missions  (f)  =  8 
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"ilssion  hours  (d)  ~  2 
number  of  equipments  (N)  •  18 

3.  Develop  oiathematical  techniques  for  relating  these  variables  during 
the  conceptual  phase  of  system  design  and  to  compute,  predict,  and/or  con¬ 
trol  the  manning  and  skill  requirements. 

Because  of  the  need  for  hardware  an.'^lvsis  and  operational  performance 
requirements,  a  causal  mathematical  model  approach  was  pursued.  The  mathe¬ 
matical  model  relates: 

a.  black  boxes  and  associated  repair  and  failure  rates 

b.  personnel 

c.  spare  black  boxes 

d.  waiting  time  to  m.'ike  a  black  box  available  alter  demand 

The  output  of  the  model  (waiting  time)  is  rel.ated  to  the  sv8t«‘m  opera¬ 
tional  requirements  and,  through  application  of  an  algorithm,  allows  the 
achievement  of  minimum  manning  for  specified  system  operational  requirements. 

In  addition,  a  method  was  developed  which  will  allow  for  scheduling  of 
training-manpower  resources  in  which  consideration  is  taken  of  the  following: 

a.  training  requirements  in  terms  of  time  necessary  to  achieve  a 
specified  skill 

b.  phasing-in  of  new  systems  and  the  concomitant  demands  on  avail¬ 
able  skills 

c.  phasing-out  of  old  systems  and  the  concomitant  availability  of 
skills 

d.  the  manpower  phasing  into  and  out  of  |»ers«>nn«‘l  inventory  bi-cause 
of  enlistment  and  discharge 

FINDINGS 

1.  It  is  anticipated  that  the  manning  prediction  technique  in  this  re¬ 
port  does  not  differ  radically  from  intuitive  procedures  presently  used  by 
management  personnel  in  the  Air  Force.  The  recognized  differences  lie  in 
the  formal  structure  of  the  manning  problem.  These  differences  are: 

a.  mathematical  statement  of  the  manning  goal 

b.  the  causal  relationship  between  this  goal  and  manning  through 
the  following: 

(1)  spares 

(2)  waiting  time 

2.  The  manning  objective  in  this  report  is  roathematicall /  equivalent  to 
nuximizing  the  total  operational  hours  of  the  operational  units,  given  a  fixed 
distribution  of  skill  hours. 
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3.  The  objectives  are  mathematically  equivalent  to  maximizing  total 
training  time  (of  operational  units)  for  a  given  investment  in  skill  hours. 

4.  Precise  knowledge  of  operational  requirements  may  not  be  known 
and  it  is  desired  to  know  how  the  manning  varies  as  the  operational  re¬ 
quirements  change.  This  ability  to  perform  parametric  analysis  may  be 
Important  in  comparison  of  alternative  systems  or  in  jointly  allocating 
fixed  manning  resources  to  two  or  more  systems, 

5.  The  technique  will  allow  evaluation  of  error  in  input  information 
through  cause  and  effect  relationships  developed. 

6.  Having  established  causal  relationships,  required  refinement  of 
specific  prediction  techniques  may  be  directed. 

CONCLUSION 

Operating  procedures  must  undergo  testing  and  modification  to  achieve 
a  good  usable  end  product.  For  a  prediction  procedure  such  as  the  one  given 
in  this  report,  testing  should  examine  various  key  characteristics  such  as 
relirbility  and  validltv.  A  reliable  pr(  edure  should  permit  different 
people  to  achieve  about  the  same  results,  given  the  same  inputs.  A  valid 
procedure  would  have  as  its  product,  predicted  manning  close  to  that  which 
actually  is  required  to  do  the  job.  Potential  error  sources  discussed  in 
the  report  should  be  subjected  to  evaluation  via  a  test  program  for  the 
manning  teclinique.  Analysis  of  causes  of  errors  in  predictions  could  lead 
to  modification  of  procedures  to  remedy  problem  areas. 

Test  application  of  the  procedure  would  also  provide  a  check  on  the 
clarity  and  comprehensiveness  of  the  procedures  and  would  point  out  areas 
where  rewriting  or  revision  is  required  for  ease  of  application. 
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Purvis,  R.  E.,  Validation  of  Queuing  Techniques  for  Determining 

W.  K.  System.  Hanning  and  Related  Support  Requirements 

McLaughlin,  R.  L. 


System  manning  has  been  a  persistent  problem  with  all  industry  and  the 
military  services.  It  has  been  a  particular  problem  with  the  Air  Force  be¬ 
cause  of  the  multitude  of  systems  and  personnel  skill  involved.  Tlie  mathe¬ 
matical  model  to  be  validated  here  is  based  on  finite  cyclical  queuing 
theory.  Details  of  the  model  ati.'  contained  in  a  previous  report,  "A  Queuing 
Model  for  Determining  System  Manning  and  Related  Support  Requirements." 

GENERAL  RESEARCH  APPROACH 

A  program  was  conducted  to  establish  the  validity  and  reliability  of  a 
technique  of  mathematical  modeling  for  predicting  manning  requirements  for 
weapon  systems.  The  technique  was  applied  to  two  systems:  the  F-105D  fire 
control  system  (FCS) ,  which  presently  is  operational,  and  the  C-lAl  system, 
which  is  scheduled  for  operation  in  the  near  future. 

ANALYSIS 


Before  a  decision  is  made  that  a  predictive  mathematical  model  is 
acceptable,  a  validation  is  needed.  The  ways  in  which  a  prediction  model 
may  not  meet  stated  objectives  are: 


1.  The  model  may  include  superfluous  independent  variables  which  have 
no  appreciable  effect  on  the  dependent  (output)  variable. 

2.  The  model  may  omit  .significant  independent  variables. 


3.  The  model  may  assert  an  inaccurate,  incorrect,  or  inadequate  func¬ 
tional  relation  between  the  dependent  variables.  This  may  be  the  case  for 
only  certain  ranges  of  values  of  one  or  more  of  the  independent  variables. 


4.  A  valid  model  apparently  may  fail  due  to  inaccurate  parameter  pre¬ 
dictions.  That  is,  inaccurately  predicted  values  of  an  independent  variable, 
substituted  into  the  model,  may  yield  an  incorrect  value  for  the  dependent 
variable;  whereas  accurate  predictions  would  have  yielded  an  acceptable  value. 


5.  Every  mathematical  model  of  a  real  system  is  an  approximation.  An 
extremely  accurate  approximation  is  possible  bv  taking  many  independent 
variables  into  account  and  using  accurate  values  for  these  variables.  If 
some  of  the  variables  are  neglected  because  of  poor  model  construction,  or 
because  they..are  not  measurable,  the  approximation  can  be  inaccurate.  There¬ 
fore,  it  is  necessary  to  decide  in  advance  what  degree  of  accuracy  will  lead 
to  model  acceptability.  Tn  general,  sophistication  and  refinement  of  models 
are  subject  to  diminishing  returns,  so  that  each  further  Improvement  bought 
at  the  same  cost  in  technical  effort  will  yield  a  smaller  improvement  in" 
accuracy.  Accordingly,  it  is  desirable  to  attempt  model  sophistication  only 
conun6nsurat e  with  desired  <'iccurcicy« 
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The  points  discussfd  above  form  the  fundamental  criteria  used  in  the 
validation  of  the  model  evaluated  for  manning  prediction  capability  under¬ 
taken  in  this  program. 

FIN[)INC;S 

1.  The  manning  technique  evaluated  in  this  program  has  been  demon¬ 
strated  to  be  a  valid  and  reliable  technique  for  predicting  manning  require¬ 
ments  for  weapons  systems. 

2.  The  technique  has  been  found  to  be  relatively  easy  to  apply,  and  is 
compatible  with  existing  methods  for  predicting  reliability  and  maintain¬ 
ability  parameters. 

3.  The  model  is  based  on  random  processes  approximating  field  conditions, 
and  produces  valid  results  using  state-of-the-art  reliability  and  maintain¬ 
ability  techniques. 

4.  The  prediction  for  the  FCS,  using  field  data  as  sources  for  parameter 
estimation,  yielded  good  results  when  compared  with  operational  performance 
and  existing  manning. 

5.  The  results  of  the  FCS  analysis  also  verified  the  fact  that,  as  with 
any  technique,  the  validity  of  this  technique  depends  upon  the  accuracy  of 
the  inputs. 

6.  There  is  also  evidence  in  the  program  results  that  the  technique  pro¬ 
vides  a  means  for  seeking  out  sources  of  inefficient  personnel  utilization 
and  sparing  procedures. 

7.  It  has  been  agreed  that  the  reliability  of  a  model  lies  in  the 
ability  to  produce  consistent  results  with  different  experimenters. 

8.  The  validation  program  did  aot  formally  test  for  reliability;  how¬ 
ever,  a  method  was  developed  which  establishes  a  general  procedure  for 
achieving  reliability.  This  proceuure  was  found  to  be  quite  sensitive  to, 
and  deiHMident  on,  methods  for  estimating  parameter  values  and  organizational 
structure  of  the  support  system. 

CONCLUSION 


The  model  prediction  for  the  FCS,  using  field  data  for  parameter  estima¬ 
tion,  yielded  good  results  when  compared  with  operational  performance.  More¬ 
over,  it  was  shown  that  the  operational  performance  could  be  achieved  by  34% 
fewer  personnel  than  the  manning  set  by  the  table  of  organization.  The  model 
prediction  for  the  FCS,  using  conceptual  data,  resulted  in  substantially  the 
same  manning  for  the  maintenance  shop  as  that  developed  from  the  measured 
data;  but,  because  maintenance  concepts  had  been  changed  in  the  field,  the 
number  of  flight  line  airmen  was  larger  than  the  measured  data.  The  manning 
prediction  for  the  C-141  system,  based  on  operational  rates  planned  for  the 
system  and  field  data  on  tiie  0130  system,  resulted  in  a  prediction  of  819 
airmen  in  the  organlz.it  ional  maintenance  squadron  and  476  airmen  in  field 
maintenance  squ.adron.  With  the  predicted  manning  it  cou’J  he  expecti'd  th.it 
an  oper.itlon.il  re.idiness  of  78%  could  be  maintained. 
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This  report  discusses  four  related  processes  for  developing  systems 
within  manning  and  skills  constraints.  The  point  is  made  that  the  military 
services  are  beginning  to  feel  the  need  to  exert  some  measure  of  control  over 
system  design  where  human  resources  are  involved. 

GENERAL  RESEARCH  APPROACH 


The  report  covers:  (1)  research  to  demonstrate  that  certain  human  re¬ 
sources  data,  such  as  manning  and  skills  data,  when  used  as  design  constraints 
along  with  other  constraints  does  affect  system  design;  (2)  techniques  by 
which  to  match  or  balance,  through  trade-off  practices,  the  hardware,  the 
human  and  other  system  support  or  logistic  functions  in  ordei  to  get  a  best 
mix  depending  on  the  r(*al  world  operational  goals;  (i)  a  new  specification 
by  which  personnel  requirements  are  staled  in  terms  of  system  requirements, 
for  integrating  more  fully  the  human  requirements  with  the  system  engineering 
processes;  (4)  a  newly  di^velojied  computer  based  data  haiuiling  system  for 
human  resources  data  in  the  conception,  development,  test  and  operation  of 
systems.  The  new  data  handling  system  is  also  an  integrator  of  liuman  and 
hardware  data  within  and  across  systems,  and  a  mechanism  for  comparing  field 
performance  data  with  early  design  data. 

ANALYSIS 

A  study  was  conducted: 

1.  to  determine  the  effect  on  system  design  of  using  certain  human 
factors  data,  in  this  case,  manpower  and  skills  data  as  design  requirements. 

2.  to  determine  under  i/hat  conditions  these  inputs  could  be  made  to  have 
maximum  Influence  on  system  configuration. 

The  study  simulated  the  design  of  the  i'itaii  111  I’roi>ellant  I  ransfer  and 
Pressurization  Subsystem  (PTPS)  by  gathering  tlu*  equipment  and  personiu'l 
documentation  produced  during  the  Titan  111  PTPS  development  and  distilling 
them  into  the  key  inputs  which  has  been  provided  to  the  original  designers. 

Six  highly  experienced  propellant  system  designers  were  selected  as 
subjects  and  given  the  problem  of  designing  the  PTPS,  starting  from  the 
Statement  of  Work.  As  equipment  inputs,  they  received  the  information  the 
original  designers  had  received  —  functional  requirements,  equipment  speci¬ 
fications,  and  hardware  information.  They  also  received  in  the  Statement  of 
Work  a  manpower  constraiiit  which  required  them  not  to  exceed  a  ci-rtain  crew 
size  and  skill  level.  In  addition,  they  received  incrementally  ot  lier 
personnel  resources  inputs  such  as  task  sequences,  human  error  data,  and  task 
time  limits. 

During  the  test  period,  the  engineer  subjects  were'  re'quired  to  deve’lop 
all  the  outputs  they  would  ordinarily  be  required  to  produce  in  actual  design: 
schematics,  equipment  descriptions,  operating  procedures,  control  pane'l 
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skitilus,  Jists  <jf  required  hardware.  The  detail  -i 

was  Kteat  enough  so  that  -.t  th^  ^  a  f  u  engineers  provided 

signs  could  be  evaluated  bv  math  "  i-?  testing  period  the  subsystem  de- 

inc  luding  equ;p::n^^tuab1lu^:::  s:?e\r"™^  effectiveness  measures, 

vlewed^ntl^ivartrite^minVwhetJer^^  the  engineers  were  inter- 

t.puts  were  useful  to  them  a^d^f  ^h^JaL"  L^Lt^^^h^^rd^Jg^^ 

FINDINGS 

averagi:;  i“o  Z  „°L  c  L*:.  top-level  acha„Hcs 

personnel.  '"'’ercas  the  large  crew"  configurations  required  8.3 

•ant  ‘design  approach  followed  by  an  individual  engineer  has  a  signlfi- 

enginUr^s  deslL reaction  to  the  manpower  and  personnel  skill  data.  The 
perJence  t^aTf radTtr^' developed,  based  on  his  own  past  ex- 
Lgs  thit  c  n  Piit  IV  !""temental  timing  of  the  human  factors  inputs 

3.  The  potential  influence  of  manpower  and  oersonnol  cUiii  a  *■ 
vant  terms.  contraints,  and  (b)  phrased  in  concrete  rele- 


CONGLUSION 

i'sslHESEiS'— 

test  computer  techniques  and  design  a  "pilot"  mLel  f*^*^^*^ 

ment  into  a  fiTi-T  ^  ^  model  for  subsequent  develop- 

Development  ObtecUvf  "LtoMt!rf  t  a  proposed  Advanced 

Program  Of  Ices"  MD^Pn^  Handling  for  System 

as  f  deveJoXt  P^og^!  '  '™‘‘  “  --IHaratlon 
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Maintenance  Job  Performance 
1”!!  j^ib_sy>:  t_t^^^^  *  ' 


Human  resources  requirements  and  data,  particularly  those  r-lated  t 
training,  have  little  significant  impact  upon  equipment  configuration, 
knowledge  of  equipment  design,  training,  and  experience  factors  does  not 
necessarily  provide  information  on  how  they  influence  performance.  The 
described  in  this  report  was  aimed  at  providing  information  on  how  these 
variables  are  Interrelated. 


o 

Having 

study 


GENERAL  RESEARCH  APPROACH 


The  study  was  designed  to  discover  certain  character 
design  which  affect  training  time  and  performance  level, 
poses  of  the  present  study  were  to  determine: 


Istlcs  of  equipment 
The  specific  pur- 


1.  Those  design  characteristics 
maintenance  technician  responds  to  in 


of  the  selert('d  subsystems 
performing  maintenance. 


wli  ich 


t  he 


2.  How  effective  maintenance  is  on 

3.  How  technician  effectiveness  is 
teristics. 


the  selected  subsystem. 

related  to  particular  design  charac- 


4.  What  differences  in  maintenance  performance  are  related  to  differen¬ 
tial  amounts  of  experience,  training,  and  skill  level  of  personnel. 

5.  What  parameters  of  skill  maintenance  personnel  recognize  as  dif¬ 
ferentiating  various  levels  of  skill  and  how  these  relate  to  the  3  5  7 

level  categories  used  in  the  Air  Force.  ’  ’ 


6.  Whether  the  study  methodology  is  effective  and 
which  should  be  made  to  improve  the  effectiveness  of  the 


those  modifications 
study  methodology. 


ANALYSIS 


After  the  airborne  electronic  equipments  in  the  inventory  were  reviewed 
two  autopilot  systems  were  selected  for  study. 

Subjects  for  the  study  were  those  technicians  assigned  to  autopilot 
maintenance  at  two  Air  Force  bases. 

2.  The  study  methodology  involved  direct  observations  of  technicians 
performing  troubleshooting  type  maintenance.  rinician.s 

about “ere  Interviewed 


4.  At  the  conclusion  of 
rated  all  technician  subjects 


the  study,  maintenance  supervisors  ranked  and 
on  a  series  of  specially  construct(>d  scales. 
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b.  Data  collectors  also  rated  the  major  characteristics  of  the  equip¬ 
ment  maintained.  These  ratings  were  correlated  with  indices  of  technician 
performance. 

6.  The  resultant  correlations  were  subjected  to  multiple  regression 
analysis,  and  those  factors  contributing  a  significant  amount  to  performance 
variance  were  extracted. 

As  a  possible  substitute  for  prolonged  observations,  a  subjective  report 
test  battery  was  developed  to  permit  subject  personnel  to  report  their  own 
maintenance  performance.  This  battery  was  administered  at  the  conclusion  of 
the  study  to  all  technician  subjects  and  to  two  classes  of  autopilot  mainte¬ 
nance  students. 

FINDINGS 


1.  The  performance  data  sought  to  determine  the  relationship  between 
indices  of  maintenance  behavior  and  various  classes  of  predictor  variables. 
After  the  variables  which  Influenced  maintenance  performance  were  identified, 
they  were  organized  into  factors  by  interpreting  their  interrelationships. 
These  factors  are: 

a.  Accessibility 

(1)  to  the  equipment  on  which  maintenance  must  be  performed. 

(2)  within  the  equipment  to  be  maintained. 

b.  Diagnostic  Information  represented  by 

(1)  the  information  supplied  directly  from  the  equipment  being 
maintained . 

(2)  the  information  derived  from  test  equipment  used  to  check 
out  the  equipment. 

(3)  information  from  T.O.'s  and  checklists. 

c.  Kqiiipment  structure  consisting  of 

(1)  chassis  weight  and  dimensions. 

(2)  number,  type,  and  arrangement  of  internal  components. 

d.  General  technician  capability,  particularly  for 

(1)  test  equipment  usage. 

(2)  m.i  iiil  I'n.'iiici’  liinction  ;ind  procedures. 

2.  The  data  obl.iiiicd  from  tlie  exper  iment.a  1  test  batterv  Indicate  that 
experienced  m.ii  nt  en.ince  t  eehn  i  c  i  .ins  e.in  rt'liabiy  predict  cert.ain  .aspects  ol 
ma  i  nt  enanee  ()er  f  orm.anee . 
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3.  Technicians  cun  reliably  estimate  troubleshooting  task  times,  task 
difficulty,  and  diagnostic  strategy  required  by  the  maintenance  problem. 

CONCLUSION 

Four  general  factors  were  found  to  be  primarily  responsible  for  the 
maintenance  performance  observed.  The  factors  were  accessibility,  diagnostic 
information,  equipment  structure,  and  general  technician  capability.  A 
residual  experience  factor  was  also  found.  From  the  results  of  the  admini¬ 
stration  of  the  subjective  report  test  battery,  it  seems  feasible  for  mainte¬ 
nance  technicians  to  predict  maintenance  duration  of  failure  problems  with 
which  they  are  familiar,  the  relative  difficulty  level  of  these  problems, 
and  the  order  in  which  they  troubleshoot  these  failures. 

If  the  conclusions  reached  are  to  be  accepted  with  any  confidence,  they 
must  be  replicated  with  a  larger  number  of  maintenance  personnel  and  different 
equipments.  Verification  of  the  hypotheses  raised  by  this  study  can  be  ac¬ 
complished  by  selecting  equipments  from  the  Air  Force  operational  inventory 
which  vary  systematically  in  terms  of  the  general  factors  found  and  then 
collecting  malntenai,''.e  performance  data  on  those  equipments,  making  use  of 
the  experimental  test  battery  procedure.  In  addition  to  this,  methods  must 
be  developed  for  obtaining  better  information  on  training  parameters  than 
current  Air  Training  Command  records  provide. 
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Hopper,  Raymond,  Training  Difficulty,  and  Job  Performance 

Potempa,  Kenneth  W. 

Ihe  human  resources  of  the  Air  Force  have  a  major  impact  on  the 
operational  capabilities  and  overall  costs  of  systems.  However,  information 
on  human  resources  requirements  is  generallv  introduced  late  or  not  at  all 
into  system  design.  Previous  studies  by  the  Air  Force  Human  Resources 
Laboratory  indicate  such  information  does  affect  design  if  it  is  available 
in  suitable  quantitative  form  early  In  design  and  if  the  Statement  of  Work 
requires  that  it  be  used.  This  study  investigated  the  relationship  between 
avionics  subsystem  design  characteristics  and  training  time,  training  cost, 
and  job  performance. 

(;f.ner/\l  rfsf./\rch  approach 

A  list  of  design  variables  believed  to  affect  training  and  job  perform¬ 
ance  was  established  and  supplemented  with  personnel  variables.  Twenty-nine 
measures  of  the  equipment  design  were  taken,  ranging  from  the  number  of  test 
points  to  the  number  of  throw-away  components.  The  personnel  variables  used 
were  aptitude  test  scores  and  amount  of  training  and  experience.  Thirty 
functional  loops,  from  ten  avionics  subsystems,  were  selected  as  the  units 
of  equipment  to  be  investigated.  Functional  checkout  procedures  were  identi¬ 
fied  or  constructed  for  each  functional  loop.  Ten  Air  Force  (AF)  students 
were  observed  performing  each  of  the  functional  checkouts;  performance  time, 
Technical  Order  (T.O.)  reading  time,  and  errors  were  recorded.  Training 
time  and  training  equipment  cost  data  were  collected  for  each  loop.  Regression 
analyses  and  factor  analysis  were  used  to  analyze  the  results  and  derive 
equations  to  predict  training  time,  training  equipment  cost,  and  job  perform¬ 
ance  time  and  errors  from  equipment  characteristics. 

AmYSJS 

A  functional  loop  in  this  study  is  defined  as  a  network  of  circuits  and 
e(iuipmciU  :uiiLs  witliin  an  avionics  subsystem  through  which  signals  are  pro¬ 
cessed  to  perlorm  a  s|)ecific  function- 

I  lie  principal  requirements  for  a  correlational  study  of  equipment 
characteristics  influencing  maintainability  can  be  summarized  as  follows: 

1.  Establish  a  comprehensive  and  valid  listing  of  design  characteristics 
for  inclusion  in  the  study. 

2.  Sample  across  classes  of  equipment  so  that  the  results  of  the  study 
can  be  generalized  to  a  range  of  practical  equipment  items. 

3.  Collect  as  comprehensive  a  data  base  as  possible  using  the  best 
me  t  hod  s  a va liable. 

Ihe  )’,c‘neral  approach  for  this  studv  was  designed  with  these  criteria  in  mind, 
lo  iiu'i't  time  aiul  cost  limitations  in  this  probe,  restrictions  tiad  to  be 
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imposed,  but  these  were  ehosen  to  maximize  the  data  base  and  the  )>,eiu‘ra I  i za- 
bility.  The  items  of  equipment  which  were  investigated  included  navigation, 
flight  control,  communications,  and  fire  control  systems  (30  functional  loops 
within  10  different  avionics  subsystems). 

Thus  far,  the  most  satisfactory  method  of  collecting  maintenance  perform¬ 
ance  data  is  observation  by  observers  who  are  familiar  with  the  tasks  they 
are  scoring.  The  data  base  therefore,  for  this  report,  consisted  of  on-site 
real-time  observations  of  performajice  by  experienced  data  collectors. 

The  major  task  was  the  functional  checkout  procedure  since  a  complete 
functional  checkout  can  be  specified  for  each  functional  loop  of  the  equip¬ 
ment,  and  the  functional  checkout  is  the  first  stage  of  troubleshooting.  The 
checkout  verifies  that  the  reporter^  malfunction  is  indeed  present,  that  it 
is  in  the  loop  under  investigation,  and  within  that  loop  identities  the  area 
of  malfunction. 

The  study  was  carried  out  in  tl>ree  broad  phases:  first,  determine  ( lit> 
design  characteristics  to  be  included  and  select  subsystt'ms  and  tiinelionnl 
loops  to  represent  those  characteristics;  second,  collect  data;  and  third, 
analyze  results. 

FINDINGS 


1.  Four  prediction  equations  were  developed  through  the  use  of  stepwise 
regressions. 

2.  Ninety-four  per  cent  of  the  variance  in  maintenance  task  performance 
time  can  be  accounted  for  by  the  complexity  of  the  subsystem,  the  number  of 
steps  in  the  maintenance  task,  the  reliability  of  the  test  equipment  used, 
and  the  ruggedness  of  the  components  being  repaired. 

3.  Ninety-four  per  cent  of  the  variance  in  task  errors  was  found  to  be 
due  to  the  number  of  steps  in  the  pairitenance  task;  the  number  of  special 
conditions,  such  as  cooling,  that  are  required;  extent  to  which  the  T.O.  uses 
standard  symbology;  the  amount  of  training;  the  number  of  dependent  remotely 
located  components;  and  the  reliability  of  the  test  equipment. 

4.  Eighty-one  per  cent  of  the  variance  in  training  time  required  for  a 
task  could  be  accounted  for  by  the  convenience  of  test  point  location,  the 
length  of  the  task,  whether  T.O.  change  data  are  clearly  presented  and  the 
number  of  dependent  remotely  located  components. 

5.  Seventy-seven  per  cent  of  the  variance  in  the  cost  of  training  equip¬ 
ment  could  be  accounted  for  by  the  extent  to  which  maintenance  tasks  are 
automated,  the  per  cent  of  identical  circuits  used  and  the  T.O.  time. 

6.  A  factor  analysis,  using  the  orthogonal  components  method,  resulted 
in  identifying  6  factors  which  account  for  a  major  part  of  the  variance  in 
the  20  independent  variables  which  correlate  significantly  with  performance. 
Names  assigned  to  the  6  factors  are; 
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a.  Length  of  Checkout  Procedure 

b.  Equipment  Complexity 

c.  Difficulty  of  Checkout  Steps 

d.  Nonautomatic  Checkout 

e.  Diagnostic  Information 

f.  Clarity  of  Information 


.  7.  Stepwise  regressions  of  the  performance  variables  on  the  factors 
resulted  in  prediction  equations  with  multiple  correlation  coefficients  of 
0.92  or  greater. 


CONCLUSION 


The  magnitudes  of  the  regression  coefficients  establish  this  as  a  very 
promising  approach  to  deriving  human  resources  data  for  systems  under  develop- 
been  shown  to  be  true  across  a  variety  of  avionics  equipment, 
Including  communications  sets,  autopilots,  air  data  computers,  and  fire  con- 
rol  systems.  The  degree  of  predictive  accuracy  attained  in  this  study  cannot 
be  expected  to  remain  as  high  during  replications  of  this  study.  However,  the 
results  of  this  study  Indicate  that  even  with  normal  shrinkage  the  predictive 
capability  of  the  equations  will  remain  highly  significant. 
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The  human  resources  of  the  Air  Force  (AF)  have  a  major  impact  on  the 
operational  capabilities  and  overall  costs  of  systems.  However,  Information 
on  human  resources  requirements  is  generally  introduced  late  or  not  at  all 
into  system  design.  Previous  studies  by  the  Air  Force  Human  Resources  Labora¬ 
tory  indicate  such  information  does  affect  design  if  it  is  available  in  sui 
able  quantitative  form  early  in  design  and  if  the  Statement  of  Work  requires 
that  it  be  used.  This  study  investigates  the  relationship  between  avionics 
subsystem  design  characteristics  and  task  difficulty,  personnel  skill  char¬ 
acteristics,  and  job  performance. 


GENERAL  RESEARCH  APPROACH 

The  relationships  among  subsystem  design  characteristics,  personnel 
skill  characteristics,  and  job  performance  were  investigated  for  avionics 
subsystems.  A  list  of  design  characteristics  was  established,  based  on  ex¬ 
pert  opinions  of  avionics  engineers.  Air  Force  supervisors,  and  instructors. 
Functional  loops  and  line  replaceable  units  were  selected  from  ten  sub¬ 
systems  representing  navigation,  flight  control,  communications 
trol  subsystems.  Experienced  maintenance  supervisors  identified  high  sKiii 
and  low  skill  maintenance  personnel.  The  same  supervisors  associated  Per¬ 
formance  times  and  error  probabilities  with  these  personnel  for  three  mainte¬ 
nance  tasks  -  an  easy  task,  a  difficult  task,  and  a  complete  functional 
checkout  task.  Supervisors  also  rated  each  task  on  a  scale  of  difficulty. 

Both  stepwise  regression  and  factor  analysis  were  used  to  derive  equations 
for  predicting  performance  time,  error  probability,  and  design  characteristics 
Similar  analyses  derived  equations  relating  performance  time  and  errors  to 
personnel  characteristics.  Personnel  factors  of  Experience,  Aptitude,  Breadth 
of  Skills,  Motivation,  Training,  Time  in  Grade,  and  non-Air  Force  Technical 
Education  were  identified. 


ANALYSIS 

This  study  used  a  correlational  approach  to  derive  equations  for  pre¬ 
dicting  performance  times,  errors,  and  task  difficulties  from  equipment  e 
sign  characteristics  and  personnel  characteristics.  Organizational  leve 
maintenance  data  were  collected  on  27  functional  loops  from  10  avionics  sub¬ 
systems  (a  functional  loop  is  defined  as  a  network  of  circuits  and  equipment 
units  within  an  avionics  subsystem  through  which  signals  are  processed  to 
perform  a  specific  function).  Intermediate  level  maintenance  data  were  col  ■ 
lected  on  28  line  replaceable  units  (LRU's)  from  the  same  10  avionics  sub¬ 
systems  . 

Twenty-eight  (for  organizational  level  maintenance)  and  29  (for  inter¬ 
mediate  level  maintenance)  equipment  design  characteristics,  such  as  number 
of  test  points  and  accessibility  of  components,  were  measured  objectively 
from  the  equipment  itself  or  from  the  Technical  Order  (T.O.).  Where 
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obU-cLive  measurement  was  not  possible,  rating  scales  were  completed  by  AF 
personnel.  Sixteen  personnel  variables,  such  as  aptitude  scores,  amount  of 
training,  and  military  rank,  were  used. 

Three  maintenance  tasks  were  identified  for  each  functional  loop  and 
LRU  an  easy  task,  a  difficult  task,  and  a  functional  checkout  task.  For 
each  task,  experienced  AF  maintenance  supervisors  estimated  performance  times 
and  error  probabilities  for  their  high  skill  and  their  low  skill  personnel. 
Supervisors  also  rated  each  task  on  a  scale  of  difficulty. 

The  design,  personnel,  performance,  and  difficulty  variables  were  inter- 
correlated,  and  were  entered  into  regression  and  factor  analyses.  Equations 
were  derived  to  predict  performance  and  task  difficulty  from  design  character¬ 
istics,  and  to  predict  performance  from  personnel  variables  and  task  diffi¬ 
culty.  Personnel  factors  and  design  factors  were  identified.  Personnel 
profiles  were  developed  for  high  and  low  skill  groups.  Plots  of  performance 
versus  difficulty  and  task  completion  versus  time  were  developed. 

FINDINGS 


1.  The  multiple  correlation  coefficients  (Rs)  for  predicting  times 
from  design  characteristics  ranged  from  0.87  for  high  skill  personnel  per¬ 
forming  intermediate  maintenance  to  0.93  for  low  skill  personnel  performing 
intermediate  maintenance,  indicating  that  from  75  to  86%  of  the  observed 
variance  in  performance  times  can  be  accounted  for  by  as  few  as  2  or  as  many 
as  8  predictor  variables  such  as  number  of  steps  in  checkout  procedure, 
accessibility  of  components,  use  of  identical  circuits,  and  level  to  which 
checkout  is  carried  (LRU,  module,  or  component).  The  Rs  for  the  error 
equations  ranged  from  0.59  for  high  skill  personnel  performing  organizational 
maintenance  to  0.85  for  low  skill  organizational  maintenance,  with  1  to  4 
predictor  variables  accounting  for  36  to  72%  of  the  observed  variance  in 
error  prooabilitles. 

2.  The  equations  for  predicting  task  difficulty  of  functional  checkout 
procedures  from  equipment  design  characteristics  gave  Rs  of  0.73  at  organiza¬ 
tional  level  (53%  of  the  variance)  and  0.80  at  intermediate  level  (64%  of 

the  variance),  from  4  predictors. 

3.  Regression  equations  for  predicting  performance  times  from  task 
difficulty  and  1  or  2  personnel  characteristics  yielded  Rs  of  0.50  for  high 
skill  intermediate  maintenance  to  0.73  for  high  skill  organizational  mainte¬ 
nance  (25  t)  53%  of  the  variance).  The  corresponding  equations  for  error 
probabilit  i''s  from  3  to  5  predictors  yielded  Rs  from  0.59  for  high  skill 
organizational  maintenance  to  0.78  for  low  skill  organizational  maintenance 
(36  to  62%  of  the  variance). 

4.  Factor  analyses  on  the  performance  and  personnel  variables  isolated 
tactors  of  Length  of  Service,  Aptitude,  Experience,  Motivation,  Air  Force 
Technical  Training,  and  lime  in  Cratle,  at  tlie  organizational  level.  At  the 
intermediate  level,  the  factors  were  Length  of  Service,  Aptitude,  Electronic 
Aptitude,  Avionics  Experience,  and  non-Air  Force  Technical  Training. 

5.  Factor  analyses  of  design  variables  indicated  that  factors  associated 
with  maintenance  performance  times  and  error  probabilities  were  Checkout 
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Complexity,  Checkout  Information,  Length  of  Checkout,  Accessibility,  Equip 
ment  Complexity,  and  Test  Equipment  and  Adjustments. 


6.  Personnel  profiles  for  the  16  personnel  variables 
for  each  subsystem  and  for  each  maintenance  level;  at  organ 
intermediate  maintenance  levels  the  low  skill  profiles  have 
times  the  error  probability  of  the  high  skill  profiles. 


were  developed 
Izational  and  at 
approximately  6 


7.  Plots  of  time  and  errors  as 
accelerated  relationships.  High  and 
errors  than  in  performance  times. 


functions  of  difficulty  level  indicate 
low  skills  are  more  clearly  separated  in 


CONCLUSIONS 

The  .  ults  of  the  present  study  should  be  replicated  and  extended  across 
additional  subsystems  and  types  of  avionics  equipment. 

measurement  methods  should  be  validated,  and  a  better  technique  for  dete 
mining  error  probability  should  be  developed.  The  relationship  between  tas 
difficulty  and  the  performance  measures  should  be  further  investigated  an 

clarified. 
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Crossman,  Edward  R.F.W. ,  The  Impact  of  Technological  Change  on  Manpower 
Laner,  Stephen  and  Skill  Demand:  Case-Study  Data  and  Policy 

Implications 

The  research  in  this  report  represents  an  expansion  and  development  of 
earlier  work  by  the  present  authors  reported  in  October  1966  under  the  title 
"Evaluation  of  Changes  in  Skill  Profile  and  Job  Content  Due  to  Technological 
Cliange:  Methodology  and  Pilot  Results  from  the  Banking,  Steel  and  Aerospace 

Industries." 

GENERAL  RESEARCH  APPROACH 


The  overall  objective  of  this  report  was  to  analyze  manpower  and  skill 
impact  of  technological  change.  The  specific  objectives  were: 

1.  Apply  the  skill-profile  methodology  developed  in  the  course  of 
previously  reported  pilot  Investigations  to  direct  labor  in  further  types  of 
technological  change,  thereby  enlarging  the  available  data  base  and  permitting 
more  positive  conclusions  to  be  reached. 

2.  Extend  the  methodology  and/or  modify  it  to  obtain  parallel  data  for 
Indirect  labor  in  a  subset  of  the  processes  studied. 

i.  Provide  an  overview  of  the  manpower  demand  pattern  indicated  by  the 
data  and  draw  conclusions  relative  to  manpower  policy. 

ANALYSIS 


1.  Outline  of  the  "Skill  Profile"  method 

The  "Skiil  Profile"  is  the  distribution  of  total  man-hours  required  to 
produce  a  unit  product  (or  service)  along  the  scale  from  least  to  most  highly 
skilled  labor.  A  suitable  basis  for  the  construction  of  such  skill  scales 
has  been  established  using  skill  factor  scores  derived  from  job  evaluation 
schemes  used  in  most  industries.  Comparison  between  skill  profiles  of  newer 
and  older  processes  permits  statistically  controlled  conclusions  to  be  drawn 
concerning  the  impact  of  technological  change  on  manpower  and  skill  require¬ 
ments,  provided  that  precautions  are  taken  to  control  the  effects  of  the 
following  factors: 

a.  processes  to  be  compared  must  be  matched  for  the  nature  and 
quality  of  their  material  inputs  and  outputs. 

b.  they  must  be  in  a  steady  state  to  avoid  contamination  by 
transiently  increased  man-hour  and/or  skill  requirements  due  to 
proi'css  development,  debugging,  etc. 

c.  organizational  differences  between  firms  must  be  controlled  by 
replicating  each  comparison  in  at  least  two  firms. 

Paired  comparisons  of  the  manpower  and  skill  demand  pattern  before  and 
after  a  specific  technological  advance  can  be  made  fully  quantitative  by  the 
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statistical  technique  of  analysis  of  variance.  The  effect 

terest  in  the  present  study  was  the  (skill  level  x  technology) 

which  was  tested  for  significance  against  the  residual  variance  in  the  (sk 

level  X  technology  x  firm)  interaction  term.  When  the  variance  ratio  was 

significant  at  a  selected  confidence  level,  It  was  Inferted  that  ^^e  observed 

change  in  skill  profile  had  been  caused  by  the  technological  change  studied. 

2.  Selection  of  further  direct-labor  case  studies 

The  skill  profile  method  previously  developed  was  applied  in  further 
direct-labor  studies  on  technological  changes  in  electricity  generation,  oil 
refineries,  air  separation,  and  airline  passenger  reservation  systems.  As 
before,  these  cases  were  each  selected  as  being  typical  of  a  large  class  of 
similar  technological  changes  in  the  same  and  other  industries.  ^  ' 

a  fairly  recent  and  pronounced  technological  advance  had  become  stabilized 
and  was  deemed  likely  to  replace  the  older  technology  in  the  near  future. 


3. 


Extension  of  the  "Skill-Profile"  method  to  indirect  labor 


A  criterion  of  "Process  Relatedness"  was  adopted  to  determine  or  identify 
the  indirect  labor  force.  Included  were  those  functions  whose  withdrawal 
would  cause  a  rapid  decay  in  process  performance.  Indirect  labor  was  traced 
and  allocated  to  unit  production  with  the  inclusion  of  maintenance  and  first 
level  supervision,  planning  and  scheduling,  and  the  exclusion  of  technical 
and  research  personnel  and  higher  management. 

Although  difficulties  were  encountered  in  acquiring  data  for  indirect 
labor  skill  profiles,  the  feasibility  of  applying  the  skill  profile  method 
was  established,  and  complete  (direct  plus  indirect  labor)  profiles  were 
obtained  for  a  sizeable  subset  of  the  total  case-study  complement. 

FIM»INGS 


1.  Labor  Productivity 

a.  Labor  productivity  Increased  in  72%  of  cases,  and  remained  the 
same  in  28%;  direct  labor  productivity  was  not  reduced  by  any  of 
the  technological  changes. 

b.  Greatest  gains  were  recorded  where  the  prechange  technology  re 
quired  the  greatest  amount  of  direct  human  intervention  in  t  e 
process. 

c.  In  most  cases  (88%)  inclusion  of  indirect  labor  data  reduced  the 
productivity  gains  computed  from  direct  labor  alone. 

d.  There  was  substantial  variance  between  organizations  due  to 
identical  (or  near  identical)  technological  changes. 

2.  Changes  in  Skill  Profile 

a.  The  limited  amount  of  indirect-labor  data  that  could  be  acquired 
within  the  resources  did  not  permit  statistically  controlled 
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comparisons  between  pre-  and  post-change  skill  profiles  for  the 
total  process  related  work  force. 

b.  Mean  direct  labor  skill  levels  increased  in  50%  of  comparisons, 
remained  unchanged  in  33%,  and  declined  in  17%. 

c.  Direct  labor  data  results  support  the  view  that  technological 
advance  tends  to  Increase  skill  demand. 

d.  On  the  average,  technological  change  tends  to  decrease  the  skill 
demanded  on  the  process-related  work  force  in  manufacturing  and 
service  industry. 

e.  Skill  Increases  observed  in  the  direct  labor  force  were  largely 
cancelled  out  by  decreases  at  indirect  labor  level. 

f.  Increases  in  overall  skill  level  are  associated  with  technologi¬ 
cal  advance  only  in  the  service  sector,  where  labor  demands  are 
relatively  low. 

3.  Impact  of  Technological  Change  on  Work  Force  Educational  Levels 

With  only  one  exception,  the  newer  technology  required  a  better  educated 
labor  force  than  the  old.  The  general  nature  of  the  process  had  more  effect 
than  the  technologic"'  change  in  itself. 

CONCLUSION 

It  has  become  clear  from  the  studies  completed  on  a  pilot  basis  that 
direct  labor  data  taken  alone  yield  a  misleading  picture  of  the  overall 
manpower  and  skill  Impact  of  a  given  technological  change.  It  is  necessary 
to  emphasize  conclusions  based  on  total  data  where  these  are  available,  even 
t  lunigii  for  lack  of  replication  they  were  not  tested  for  statistical  signifi- 
lance.  FurtlnT  researcit  effort  would  be  needed  to  round  out  and  substantiate 
the  present  conclusions  by  completing  the  18  comparisons  undertaken. 

Other  factors  were  not  fully  investigated  within  this  reports  Of  these, 
the  most  important  is  probably  the  relative  effectiveness  of  manpower 
utilization  across  processes  and  technologies. 
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Askren,  William  B. , 
Korkan,  Kenneth  D. 


A-13 


Design  Option  Decision  Trees;  A  Method  for 
Relating  Human  Resources  Data  to  Design  Alternatives 

The  human  resources  of  the  Air  Force  have  a  major  Impact  on  the  opera¬ 
tional  capabilities  and  overall  costs  of  aerospace  systems.  One  of  the  re¬ 
search  objectives  of  the  Air  Force  Human  Resources  Laboratory  is  to  develop 
methods  of  incorporating  human  resources  data  (HRD)  in  the  decisions  which 
define  system  design  characteristics.  This  study  was  performed  to  determine 
the  feasibility  of  predetermining  the  design  options  available  to  the  engi¬ 
neer  as  he  progresses  through  a  design  problem. 

GENERAL  RESEARCH  APPROACH 


It  would  be  most  useful  if  a  method  could  be  developed  for  relating  human 
resources  data  to  detailed  design  features.  These  data  could  then  enter  into 
decisions  regarding  selection  of  specific  design  options.  The  purpose  of  this 
investigation  was  to  determine  the  feasibility  of  Identifying  the  design 
options  available  to  the  engineer  as  he  progresses  through  a  design  problem. 
The  approach  involved  three  steps: 

1.  Design  options  available  to  the  engineer  in  the  propulsion  and 
flight  control  subsystem  areas  were  identified  and  placed  in  decision  tree 
form. 


2.  Design  options  and  the  concept  of  arranging  the  options  in  decision 
tree  form  were  evaluated  by  eight  engineers  experienced  in  design  of  these 
classes  of  subsystems. 

3.  The  information  and  data  obtained  from  the  engineer  evaluators  were 
analyzed  (after  personal  interviews),  and  new  decision  trees  of  the  propulsion 
and  flight  control  areas  were  prepared  to  represent  a  composite  of  their 
recommendations. 

ANALYSIS 

Two  experienced  engineers  worked  together  to  produce  five  first  genera¬ 
tion  decision  trees  describing  propulsion  and  flight  control  subsystems. 

Eight  engineers  were  chosen  as  evaluators;  four  with  experience  in  propulsion, 
and  four  with  experience  in  flight  control.  Each  evaluator  was  given  a  set 
of  the  decision  tree  drawings  related  to  his  area  of  experience  and  asked  to 
study  and  critique  them.  After  a  few  days,  an  interview  was  held  with  each 
of  the  evaluators.  During  the  Interview  the  engineer  was  questioned  con¬ 
cerning  his  opinion  of  the  validity  of  the  design  options  and  the  flow  and 
nature  of  decisions  represented  in  the  tree.  He  was  also  asked  for  his 
evaluation  of  the  completeness  of  the  tree  with  respect  to  the  design  options 
available  in  the  subsystem  area,  and  the  feasibility  and  practicality  of 
representing  the  available  options  in  a  decision  tree  format.  And  finally, 
he  was  asked  to  recommend  changes  to  the  tree  in  order  to  represent  the  de¬ 
sign  decision  proces.-  according  to  his  experience.  The  information  and  data 
obtained  from  the  evaluators  were  analyzed,  and  new  decision  trees  of  the 
propulsion  and  flight  control  areas  were  prepared  to  represent  a  composite  of 
their  recommendations. 
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FJ^INGS 

1.  Feasibility  and  Practicality  of  Preparing  Design  Option  Decision 

Trees 

It  was  found  to  be  feasible  and  practical  to  prepare  design  option 
decision  trees,  to  identify  the  major  disciplines  or  subsystems  of  an  aero¬ 
nautical  system,  and  to  further  determine  the  delineation  of  design  options 
relevant  to  the  subsystems. 

2.  Validity  and  Thoroughness  of  Design  Option  Decision  Trees 

All  eight  evaluators  judged  that  the  originally  prepared  decision 
trees  were  valid  representations  of  the  design  decision  process,  and,  although 
not  complete,  adequately  demonstrated  the  concept.  The  second  generation 
trees  were  developed  after  recommended  expansions  and  changes  were  given  by 
the  evaluators.  Although  design  option  decision  trees  can  be  made  initially 
utilizing  one  or  two  people,  it  is  necessary  and  feasible  to  incorporate  the 
consensus  of  a  number  of  experienced  engineers  to  develop  the  final  concepts. 

3.  Relating  Human  Resources  Data  to  Design  Options 

All  eight  evaluators  judged  that  the  design  options  should  be  meaning¬ 
fully  evaluated  for  human  resources  factors  such  as  maintenance  difficulty, 
personnel  skill,  training  difficulty  and  manpower  costs,  as  well  as  for 
engineering  parameters  such  as  reliability,  development  costs,  performance, 
and  weight. 

4.  Computer  Processing  of  Design  Option  Decision  Trees 

Recently  developed  computer  software  allows  the  storage  and  retrieval  of 
tree  forms  of  data.  It  is  therefore  highly  likely  that  design  option  decision 
trees  can  be  processed  by  computer.  Computer  software  must  be  expanded  to 
include  the  capability  of  storing  the  human  resources  implications  of  the 
design  options.  This  would  allow  the  design  engineer  to  work  through  a  de¬ 
sign  problem  at  the  computer  console  with  computer  memory  providing  the  human 
resources  (or  engineering)  implications  of  the  design  alternatives  under 
consideration. 

5.  Other  Uses  of  Design  Option  Decision  Trees 

A  number  of  other  applications  of  the  decision  tree  format  for  design 
information  were  identified: 

a.  The  decision  tree  format  clearly  indicates  the  number  of  choices 
available  to  the  engineer  at  each  decision  node. 

b.  It  shows  the  interrelationships  of  decisions. 

c.  The  format  allows  for  describing  advancements  to  the  state-of- 
the-art  at  the  decision  points. 

d.  Tt  provides  a  priority  sequence  of  the  decisions. 
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CONCLUSION 


The  preparation  of  design  option  decision  trees  for  propulsion  and 
flight  control  subsystems  was  found  to  be  feasible  and  practical.  Although 
design  option  decision  trees  can  be  made  initially  utilizing  one  or  two 
people,  it  is  necessary  and  feasible  to  incorporate  the  consensus  of  a 
number  of  experienced  engineers  to  develop  the  final  concepts.  Design  options 
can  be  meaningfully  evaluated  for  human  resources  factors  such  as  maln;.enance 
difficulty,  personnel  skill,  training  difficulty  and  manpower  costs,  as  well 
as  for  engineering  parameters  such  as  reliability,  development  costs,  per¬ 
formance,  and  weight.  It  was  also  determined  that  the  design  option  decision 
trees  can  be  processed  by  computer.  This  would  allow  machine  storage  and 
retrieval  of  the  many  design  options  and  the  related  human  resources  data 
which  could  be  used  to  evaluate  each  option. 
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Askren,  William  B. ,  Human  Resources  Sensitivity  to  System  Design 

Korkan,  Kenneth  D. ,  Trade-off  Alternatives;  Feasibility  Test  with 

Watts,  George  D.  Jet  Engine  Data 

Historically,  skilled  maintenance  personnel  were  provided  for  Air  Force 
weapon  systems  after  the  hardware  was  delivered  to  the  operational  command. 
Maintenance  personnel  selection  and  training  was  accomplished  as  a  form  of 
reaction  to  the  demands  of  the  equipment.  Today,  because  of  system  complexity 
and  the  faster  pace  of  events,  human  resources  planning  is  characterized  by 
analyses  which  predict  maintenance  manpower  needs  early  in  system  development 
life.  This  allows  personnel  selection  and  training  to  begin  before  the  hard¬ 
ware  is  delivered  to  the  field. 

GENERAL  RESEARCH  APPROACH 


The  feasibility  of  developing  Design  Option  Decision  Trees  to  a  level  of 
detail  which  shows  hardware  involved  in  "on-aircraft"  maintenance,  and  the 
feasibility  of  measu’'ing  the  sensitivity  of  human  resources  data  to  design 
trade-off  problems  depicted  in  these  trees  were  investigated.  The  approach 
included  expanding  a  portion  of  an  earlier  developed  aircraft  jet  engine  tree, 
selecting  trade-off  problems  from  the  expanded  tree  for  sensitivity  analysis, 
and  collecting  psychometric  data  from  experienced  jet  engine  mechanics  re¬ 
garding  this  sensitivity.  Five  Design  Option  Decision  Trees  were  developed 
for  turbofan  jet  engines.  Eight  turbofan  trade-off  problems  were  evaluated 
for  effect  on  human  resources.  The  factors  of  training  and  experience,  and 
amount  of  troubleshooting  time  (as  affected  by  choice  of  design  option);  ease 
of  maintenance;  complexity  of  tools  and  equipment;  and  crew  size  and  job 
specialty  were  examined. 

ANALYSIS 


1.  A  portion  of  the  aircraft  propulsion  Design  Option  Decision  Tree 
developed  in  the  earlier  study  was  selected  for  expansion. 

2.  An  attempt  was  made  to  develop  turbofan  jet  engine  Design  Option 
Decision  Trees  down  to  a  level  of  detail  depicting  the  smallest  item  of  hard¬ 
ware  replaced  during  "on-alrcraft"  maintenance.  This  is  comparable  to  the 
maintenance  four-digit  work  unit. 

3.  A  number  of  design  option  trade-off  problems  representing  a  variety 
of  jet  engine  characteristics  wcrr  selected  for  sensitivity  analysis  as  to 
their  effect  on  Human  Resources  Data. 

4.  Psychometric  data  were  collected  from  experienced  jet  engine  mechanics 
as  a  means  of  measuring  the  human  resources  sensitivity.  Data  were  obtained 
regarding  the  effect  of  the  design  options  on  maintenance  crew  size,  mainte¬ 
nance  job  specialties,  training  and  experience,  troubleshooting  time,  ease  of 
maintenance,  and  complexity  of  tools  and  equipment. 

FINDJ-NGS 


1.  It  was  found  feasible  and  practical  to  prepare  Design  Option  Decision 


Trees  detailed  to  the  maintenance  four-diRit  work  unit  level  of  detail  for 
turbofan  jet  engines. 

2.  Five  trees  were  developed  and  Include  a  top-level  drawing  that  shows 
the  relationship  of  turbofan  jet  engines  to  the  propulsion  subsystem. 

3.  Four  trees  were  developed  that  describe  the  design  options  for  jet 
engine  assembly,  jet  engine  installation,  and  jet  engine  supporting  systems. 

4.  The  levels  of  detail  in  the  trees  are  sufficiently  varied  (such  as 
choice  of  type  of  fan,  choice  of  type  of  starter  system,  choice  of  type  of 
compressor  bearings)  to  provide  trade-off  problems  with  a  variety  of  hardware 
characteristics. 


5.  Eight  trade-off  problems  were  selected  for  study  regarding  effect 
on  hi^an  resources.  Six  experienced  jet  engine  maintenance  mechanics  eval- 
uated  the  design  options : 


a.  The  results  show  that  the  amount  of  training  and  experience  re¬ 
quired  for  troubleshooting  a  failure  is  significantly  affected 
by  choice  of  design  options  in  six  of  the  eight  problems,  with 
as  much  as  60%  less  training  and  experience  being  required  for 
the  favored  options. 

b.  Amount  of  time  required  to  troubleshoot  a  failure  is  signifi¬ 
cantly  affected  by  choice  of  design  options  in  six  of  eight 
problems,  with  as  much  as  56%  less  troubleshooting  time  required 
for  the  favored  options. 


c.  The  complexity  of  tools  and  equipment  required  to  troubleshoot  a 
failure  is  affected  in  two  of  eight  problems. 

d.  Overall  ease  of  maintenance  is  significantly  affected  by  choice 
of  design  option  in  five  of  eight  problems. 


e.  Crew  size  and  Job  specialty  required  to  troubleshoot  a  failure 
are  each  affected  in  one  problem. 


6.  The  combined  results  indicate  that  at  the  start  of  a  subsystem  design 
e  fort  the  required  design  trade-off  problems  can  be  identified  and  evaluated 
tor  Impact  on  the  human  resources. 


CONCLUSION 


Tr  I  is  concluded  that  it  is  feasible  to  develop  Design  Option  Decision 
Trees  to  the  maintenance  four-digit  work  unit  (line  replaceable  unit)  level 
°Ju  providing  a  description  of  the  broad  spectrum  of  trade-offs 

that  would  be  performed  during  a  subsystem  design.  It  is  concluded  that  it 
is  feasible  to  determine  the  sensitivity  of  human  resources  data  to  trade-off 
problems  and  design  options  using  psychometric  analysis  methods.  It  is  con¬ 
cluded  that  selection  of  a  design  option  will  influence  human  resources  and 

Flnan^^^r  ^  «“bstantial  proportion  of  design  trade-off  studies. 

Finally,  it  is  concluded  that  in  trade  studies  with  more  than  two  options  to 
consider  It  Is  highly  probable  that  an  order  of  priority  of  design  oJtlLs 
based  on  human  resources  effects  can  be  determined.  ^ 
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Tho^s,  Ralph  E.,  The  Effect  of  Various  Levels  of  Automation  on 

^ritsker,  A.  Alan,  Human  Operators'  Performance  In  Man-Machine 

Christner,  Charlotte  A.,  Systems — - 

Byers,  Richard  H. 

Many  systems  must  be  monitored  at  regular  intervals  and  maintained  within 
specified  tolerance  limits.  A  pilot,  for  example,  must  monitor  the  speed  and 
position  of  his  aircraft  and  keep  within  some  limits  of  heading  and  altitude. 
The  necessity  for  controlling  these  system  processes  arises  because  they  are 
subject  to  disturbances  both  erratic  and  constant.  In  other  words,  the 
system  process  is  subject  to  a  noise  component  superimposed  on  a  signal  which 
on  the  average,  drifts  out  of  the  specified  tolerance. 

GENERAL  RESEARCH  APPROACH 


This  report  describes  a  method  for  generating  definitive  data  on  the 
effects  of  various  levels  of  automation  on  human  operators'  performance  in 
man-machine  systems.  The  method  Incorporates  a  model  and  equipment  for 
theoretical  and  experimental  investigations.  Equipment  was  designed  and  built 
in  accordance  with  the  assumptions  of  the  automation  model  for  studying  human 
performance  in  an  automation  environment.  Functions  to  be  controlled  are 
generated  by  a  general-purpose  analog  computer. 

ANALYSIS 


The  conceptual  model  which  is  used  contains  three  essential  elements: 
the  noise  parameter,  the  drift  parameter,  and  the  tolerance  limits  for  the 
variable  to  be  controlled.  In  addition,  the  model  contains  two  essential 
human  parameters:  the  monitoring  frequency  and  the  subjective  tolerances. 
These  subjective  tolerances  are  perceptual  limits  which  may  or  may  not  be  the 
same  as  the  actual  tolerances.  They  are  included  to  permit  the  human  to 
correct  the  system  whenever  he  thinks  it  is  necessary.  He  may,  for  example, 
make  a  corrective  action  based  on  his  subjective  estimate  when  the  system 
actually  is  not  out  of  control.  These  subjective  tolerances  form  the  speci¬ 
fically  human  inputs  to  the  model,  are  variable  from  subject  to  subject,  and 
vary  from  time  to  time  in  a  given  subject.  They  are  also  subject  to  change 
due  to  motivation,  drive,  competition  or  cooperation,  fear,  fatigue,  and 
other  complex  psychological  factors.  In  addition,  these  tolerances  are  sub¬ 
ject  to  change  by  the  degree  of  uncertainty  the  subject  feels  about  his 
ability  to  control  the  system. 

The  model  allows  the  designer  to  incorporate  human  performance  properties 
into  the  control  subsystem  by  providing  methods  to  determine  the  following 
relationships:  first,  how  noise  and  drift  are  associated  with  subjective 
tolerances,  and  second,  how  complex  psychological  factors  are  associated 
with  these  tolerances.  Knowing  these  relationships,  the  designer  can  modify 
his  equipment  to  conform  to  human  requirements,  or  personnel  can  be  selected 

on  the  basis  of  the  adequacy  of  their  subjective  tolerances  to  control  a 
specific  system. 

In  the  model,  the  tasks  of  the  human  are  repetitive  cycles  of  three  types 
of  assessments  for  the  control  of  the  system.  He  must  determine  whether  or 
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not  the  system  is  in  control,  select  the  corrective  action  to  restore  or 
^intaln  control,  and  judge  the  effect  of  the  corrective  action.  The  analysis 
IS  thus  reduced  to  manageable  proportions  by  focusing  on  the  correctness  of 
the  a»eee„e„ts  aed  the  «a„„er  i„  which  they  are  made  under  var”ua  cLdiUuns. 

syatels"  describe  aim., si  all  man-machine 

Third  an  or,hef  ’r  ^  t»  Ptocitss  information. 

^  u  ^  ^  systems  aro  expected  to  perform  some  task;  that  is  to 

system  ^ach^  element  is  provision  for  control  of  the 

system.  Each  system  has  criteria  of  performance  associated  with  it  The 

Zt"of  enhancrth^atLL- 

nent  or  the  performance  criteria* 

Functions  to  be  controlled  are  generated  by  a  general-purpose  analoc 
computer.  Automatic  recording  of  input  and  response  signals  for  all  thrL 
assessment  types  is  accomplished  on  a  six-channel  recorLr.  In  additil  to 
having  separable  panels  to  permit  future  studies  of  different  organizations 

ffiJur^bv^LSr  equipment  may  be  preprogrammed  to  simulate 

ilure  by  making  certain  correction  controls  "inactive"  at  specified  times. 

ANALYSIS 

1.  The  critical  experiments  showed  that  the  equipment  is  suitahip  fnr 
conducting  automation  axparlmants.  In  addition,  the  ef?ec.a  of  paJame'er 
varlatlona  of  the  stochastic  model  yielded  results  generally  as  Lpected. 

2.  Three  pilot  experiments  were  conducted  that  involved  chanees  In 

stochastic  model,  the  level  of  automaton  th^sKjects' 
task,  and  the  assessment  cycle  time.  f>uDjects 

3.  For  each  experiment,  a  description  of  a  hypothetical,  but  reallsrin 

dTtertri  trailing  ses^i^n^ie^rco^  ’ 

ducted  to  introduce  the  automation  environment  to  the  subiect  to  pvnian- 
experl^nt.  to  describe  the  equipment  and  Its  use,  and  t^  ^I  y°  k  o' 

be  performed.  At  the  completion  of  each  experiment,  a  questlonnalrfconcer" 

Ing  the  experiment  was  answered  by  the  subjects. 

4.  In  Experiment  1,  the  subject  could  not  predict  the  behavior  ..f  thp 
relevant  system  variable  because  of  a  large  random  component  Included  In  the 
stochastic  model.  Thus,  the  decision  rules  adopted  by  the  sublectfdld  no^ 

r"r'o:Se‘rjredic!r:r-o'Sig^ifi‘:r  d  Jr 

and  machine  performance  we^e  detecjlbj:  JJrJh^v^Jlo^Js^J^jrorjrrLj”:" 

in  orL  Jo  srair^^^pjeiicribitry- rtjrr::::Lr“:r:^ 

he  levels  of  auto^tlon  for  the  Type  1  (Is  the  system  In  contj^io  .JJd  JyJ; 

aitJred'jj  i'^ne'r  Jhat 'u,r'j:j:iri”jr:J"aur: « J  ’ 

o"j  :ul:"jj:r„jje":j:-::dj" 
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6.  In  Experiment  3,  two  of  the  extreme  levels  of  automation  used  in 
Experiment  1  and  the  stochastic  model  of  Experiment  2  were  used,  and  a 
significant  difference  in  performance  scores  was  observed.  A  quantitative 
comparison  of  the  theoretical  and  observed  results  for  Experiment  3  could 
not  be  made  because  the  magnitudes  of  the  corrective  actions  were  not  suit¬ 
ably  chosen  in  this  experiment.  However,  qualitative  agreement  with  the 
model  was  fully  demonstrated. 

CONCLUSION 

The  efforts  in  this  research  program  have  produced  the  following  tangible 
results: 


1.  A  qualitative  model  in  which  the  automation  problem  is  reduced  to  its 
essential  features. 

2.  A  quantitative  model  that  Includes  two  system  parameters,  and  may 

be  analyzed  to  determine  the  assessment  cycle  times  that  will  yield  a  desired 
probability  of  control  for  given  tolerance  limits,  and  specified  magnitudes 
of  the  corrective  actions. 

3.  An  experimental  apparatus  that  is  capable  of  presenting  to  groups  of 
subjects  a  wide  variety  of  control  problems  characterized  by  the  specific 
values  used. 

A.  The  results  of  pilot  experimental  studies  that  suggest  satisfactory 
agreement  between  the  predicted  and  observed  probabilities  of  control. 
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A-16 

Buechner,  William  R.  Technological  Change  and  the  Occupational  Composition 

of  the  American  Labor  Force.  1950  -  1960 

Technological  change  is  one  of  the  most  important  factors  affecting  the 
American  labor  force,  but  as  yet  one  of  the  least  understood,  pe  kinds  of 
models  used  by  economists  to  examine  technological  change  have  been  among  the 
primary  sources  of  this  lack  of  understanding;  they  tend  to  focus  on  two-factor 
models  and  examine  questions  concerning,  e.g.,  whether  or  not  there  is  a 
labor-saving  bias  connected  with  current  technological  change.  This  report 
examines  the  effects  of  technological  change  on  the  occupational  composition 
of  the  U.S.  labor  force  between  ^950  and  1960  using  an  input-output  model, 
with  each  industry's  labor  input  coefficient  disaggregated  into  240  occupa¬ 
tional  coefficients. 

GENERAL  RESEARCH  APPROACH 


This  report  utilizes  an  input-output  model  to  measure  the  effects  of 
changes  in: 

1.  The  level  and  composition  of  final  demand 

2.  The  input-output  coefficients 

3.  Labor  productivity  in  each  industry 

4.  The  occupational  coefficients  in  each  industry  on  the  economy's 
total  employment  by  occupation. 

The  model  is  used  to  measure  the  effects  of  technological  change  on  the 
kinds  of  jobs  performed,  the  level  of  complexity,  and  their  educational  and 
training  requirements.  The  model  is  used  to  determine  whether  the  occupa¬ 
tional  substitutions  which  occurred  as  a  result  of  technological  change 
were  affected  by  changes  in  the  relative  wages  of  the  different  occupations. 

ANALYSIS 


Three  questions  were  confronted: 

1.  How  did  technological  change  between  1950  and  1960  affect  the 
economy's  demand  for  labor  from  individual  occupations?  Also,  given  the  fact 
that  technological  change  can  work  to  alter  a  number  of  different  kinds  of 
labor-demand  variables,  which  facets  of  technological  change  were  most  im¬ 
portant  in  determining  occupational  employment  trends? 

2.  Is  it  important  for  manpower  planning  to  ask  how  technological  change 
affected  the  kinds  of  work  performed  in  the  economy's  jobs,  the  content  level 
of  that  work,  and  the  educational  and  training  requirements? 

3.  Did  the  wage  changes  between  1950  and  1960  Induce  factor-substitu¬ 
tion  through  technological  change  or  was  technological  change  autonomous? 
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A  model  was  made  to  answer  these  questions.  The  model  was  a  modified 
input-output  model  developed  by  Wassily  Leontief.  It  was  assumed  that  the 
production  functions  for  the  economy's  industries  are  of  the  fixed-input- 
coefficient  form,  with  not  only  intermediate  Inputs,  but  labor  inputs  as 
well,  proportionately  related  to  output  levels.  The  model  was  Implemented 
with  input-output  data  from  the  1947  and  1958  input-output  studies  and  with 
employment  by  occupation  by  industry  matrices  from  the  1950  and  1960  Census 
9.f  Population.  Adjustments  were  made  to  the  original  data  to  correct  for  ’ 
the  different  time  periods  used  and  for  different  industry  categories  used 
in  compiling  and  reporting  the  data.  The  final  model  was  implemented  using 
65  industries  and  240  occupations. 

FINDINGS 

1.  Each  of  the  possible  sources  of  change  had  much  different  effects  on 
labor  demand,  totally  as  well  as  on  individual  occupations. 

2.  Final  demand  changes  worked  to  increase  the  demand  for  labor  from 
all  but  four  occupations;  the  average  unweighted  increase  was  33.6  per  cent 
with  very  little  variation  among  occupations. 

3.  Technological  change  worked  to  reduce  labor  demand  from  'nost  occupa¬ 
tions,  but  its  effect  was  very  diverse. 

4.  Most  importantly  for  manpower  planning,  it  appears  that  the  main 
source  of  variation  in  the  growth  rates  of  employment  by  occupations  is  to  be 
found  in  the  changes  which  individual  industries  made  in  the  occupational  mix 
of  their  labor  inputs. 

5.  Variation  in  occupational  demand  was  due  to  technological  change, 
and  more  specifically,  to  occupational  mix  changes. 

6.  Changes  in  the  final  demand  vector  were  not  significant  factors  in 
altering  the  pattern  of  work  performed  in  the  economy;  the  only  jobs  wl  ich 
experienced  much  faster  than  average  rates  of  growth  from  this  variable  were 
those  involved  with  work  in  the  fields  of  health,  education,  welfare,  and 
research  and  design. 

7.  Technological  change,  however,  did  measurably  affect  the  kinds  of 
work  performed,  reducing  employment  in  jobs  associated  with  tools  and  machine.: 
while  increasing  demand  for  labor  in  service-oriented  jobs. 

8.  Final  demand  and  technological  changes  were  both  important  factors 
in  raising  the  overall  content  of  the  economy's  jobs;  both  worked  to  increase 
the  economy  s  demands  for  jobs  Involving  high  content  levels  and  to  decrease 
demand  for  low  content  jobs. 


9.  Technological  change  over  the  decade  tended  to  increase  the  amount 
of  education  required  for  jobs,  by  Increasing  demand  for  occupations  necdiu); 
a  high  school  education  or  more  and  decreasing  demand  for  those  requi r ing  less 
than  12  years.  There  was  no  discernible  effect  cn  job  training  requirements. 

10.  Final  demand  changes  had  effects  on  educational  and  training  re¬ 
quirements  which  were  similar  to  those  of  technological  change. 
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11.  Technological  change  was  generally  factor-biased,  with  most  in- 
ustries  substituting  rising  wage  occupations  for  falling  wage  occupations. 

nf  Anticipated  wage  changes  were  a  significant  variable  in  the  choice 

CONCLUSION 

^  K  possible  to  use  an  input-output  model  to  examine  the  effects  of 

technological  change  on  the  occupational  composition  of  the  labor  force. 

include  level  and  composition  of  final  demand,  input-output 

industry,  and  the  occupational 

ficlents  in  each  Industry  on  the  economy's  total  employment  by  occupation. 
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Crossman,  Edward  R.  F.  W. ,  Evaluation  of  Changes  In  Skill-Profile  and 
Laner,  Stephen,  Job-Content  Due  to  Technological  Change; 

Davis,  Louis  E. ,  Methodology  and  Pilot  Results  From  the  Banking, 

Caplan,  Stanley  H.  Steel  and  Aerospace  Industries 

A  new  method  has  been  developed  for  measuring  the  Impact  of  selected 
advances  In  technology  on  the  distribution  of  skills  required  In  the  direct 
labor  force  In  manufacturing  and  service  Industries.  Based  broadly  on  the 
"direct  productivity"  studies  conducted  by  the  Bureau  of  Labor  Statistics 
in  the  late  forties  and  early  fifties,  the  procedure  as  applied  to  a  single 
process  entails  evaluating  the  man-hours  of  direct  labor  expended  per  unit 
product  at  each  of  several  skill  levels  and  thence  computing  skill-based 
productivity  measures  and  mean  skill  level. 

GENERAL  RESEARCH  APPROACH 


Established  job-evaluation  schemes  used  by  firms  for  deriving  basic  pay 
rates  provided  the  essential  data  for  assessments  of  skill-level ,  a  quantita¬ 
tive  skill-scale  being  obtained  by  grouping  point  scores  for  selected  factors: 
the  point  scores  for  a  single  process  fall  into  a  distribution  termed  a 
"skill-profile."  Mean  skill-levels  were  computed  from  the  skill-scale  point 
scores  multiplied  by  man-hours  per  unit  product  and  summed  over  all  workers 
employed  on  the  process.  The  standard  deviation  and  range  of  skill-levels, 
and  other  rele  ant  statistics  were  also  computed.  This  enabled  comparisons 
to  be  drawn  between  the  skill-profiles  obtained  for  two  or  more  levels  of 
production  technology. 

ANALYSIS 


Case  material  was  carefully  ^elected  and  controlled  for  the  comparisons 
to  reveal  only  the  differences  in  manpower  and  skill  requirements  due  to 
changes  in  process  technology  and  not  to  other  irrelevant  factors.  Two  (or 
more)  processes  were  matched  in  respect  of  the  dimensions,  quality  specifi¬ 
cations  and  other  relevant  characteristics  of  raw  materials  and  finished 
product.  Processes  were  in  a  steady  state,  with  a  criterion  that  two  or 
more  years'  productive  operation  had  been  applied  to  avoid  transient  effects. 

Random  error  (uncontrolled  variation)  is  needed  to  test  observed  dif¬ 
ferences  for  statistical  significance.  This  was  obtained  by  replicating 
each  matched-pair  comparison  in  a  different  firm  or  organization,  matching 
the  raw  materials  and  product  as  closely  as  possible,  and  also  technologies 
of  the  older  and  newer  processes  operated  by  the  two  (or  more)  firms  or 
organizations. 

When  controlled  data  for  two  or  more  such  paiis  of  processes  have  been 
obtained  it  becomes  possible  to  test  observed  changes  in  mean  skill-level  for 
statistical  significance  by  analysis  of  variance  of  the  man-hour  data.  Vari¬ 
ations  ("main  effects")  due  to  s'<lll-level,  technological  level  and  firm  are 
removed  first,  and  the  interaction  between  skill-level  and  t eclino logira 1 
level  is  then  tested  for  significance  against  t!i.  three-way  interaction  be¬ 
tween  skill-level,  technological  level  and  firm.  Thus,  fully  quantitative 
conclusions  can  be  drawn  from  a  small  number  of  s.ample  observ.it  ions ,  provided 
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that  a  minimum  of  four  matched  processes  are  studied  (i.e.,  two  or  more 
technological  levels  in  each  of  two  or  more  firms). 

FINDINGS 


1.  In  each  case  the  introduction  of  more  highly  mechanized  production 
methods  was  associated  with  substantial  reductions  in  total  man-hours  re¬ 
quired  oer  unit  product  (direct  labor).  Productivity  Increases  ranged  from 
26.5*  to  271Z. 

2.  A  given  process  showed  closely  similar  patterns,  both  of  unit  labor 
requirement  and  of  change  with  technological  advance,  when  operated  xn  dlf- 
ferent  firms  and  organizations* 

3.  Different  processes  showed  statistically  significant  differences  in 
skill  requirement  and  different  patterns  of  change  with  technological  advance. 

4.  The  skill-profiles  for  the  older  technologies  in  banking  and  steel 
processes  showed  a  preponderance  of  middle-level  skills,  while  in  aerospace 
machining  processes  higher  levels  predominated.  It  may  plausibly  be  sur¬ 
mised  that  the  absence  of  lower-level  skills  is  associated  with  the  fact  that 
the  older  processes  were  already  highly  mechanized. 

5.  The  reduction  in  per  unit  labor  requirements  due  to  Introduction  of 
new  processes  was  concentrated  in  the  middle  range  of  skills,  except  for  aero¬ 
space  machining  where  the  higher  level  skills  showed  most  reduction. 

6.  Mean  skill-levels  were  Increased  to  a  statistically  significant  ex¬ 
tent  in  all  cases  except  aerospace  machining  (where  they  were  reduced  signifi¬ 
cantly).  However,  the  changes  recorded  were  all  small  both  in  relative  and 
absolute  magnitude. 


7.  Absolute  per  unit  labor  requirements  in  the  skill  groups  were  reduced 
in  all  cases  except  demand-deposit  accounting  and  galvanizing.  Even  in  these 
two  instances  the  Increases  were  small  in  terms  of  full  time  job  equivalents. 
Thus,  the  evidence  does  not  support  the  widely  expressed  view  that  advanced 
production  technology  substantially  raises  the  requirement  for  highly-skilled 
operating  personnel. 

8.  The  overall  impact  of  the  advanced  technologies  studied  was  a  small 
net  Increase  in  mean  skill  level  due  to  a  larger  reduction  in  middle-level 
than  in  higher-level  skills. 

The  above  results  were  confirmed  by  assessment  of  educational  and  job- 
experience  requirements  which  were  found  to  rise  very  little  with  advancing 
technology.  However,  these  changes  could  not  be  tested  for  statistical  signlfi- 


The  diversity  of  skill-content  (as  distinct  from  skill-level) 
assessed  by  a  count  of  job-types,  and  the  results  indicate  gTilti’r 
in  one  case  only,  demand-deposit  accounting.  Job  diversity  showed 
change  in  manufacturing  Industry. 
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